Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
Authors: Yushi Hu
Weijia Shi
Xingyu Fu
Dan Roth
Mari Ostendorf
Luke Zettlemoyer
Noah A. Smith
Ranjay Krishna
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal
large language model
computer vision
reasoning
Primary area: natural_language_processing
TL;DR: we introduce Visual Sketchpad: a framework that provides multimodal LMs with the tools necessary to generate intermediate sketches to reason over tasks.

Abstract: Humans draw to facilitate reasoning: we draw auxiliary lines when solving geometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacity working memory. However, such actions are missing in current multimodal language models (LMs). Current chain-of-thought and tool-use paradigms only use text as intermediate reasoning steps. In this work, we introduce Sketchpad, a framework that gives multimodal LMs a visual sketchpad and tools to draw on the sketchpad. The LM conducts planning and reasoning according to the visual artifacts it has drawn. Different from prior work, which uses text-to-image models to enable LMs to draw, Sketchpad enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. \name can also use specialist vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment on a wide range of math tasks (including geometry, functions, graph, chess) and complex visual reasoning tasks. Sketchpad substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with Sketchpad sets a new state of the art on all tasks, including V*Bench (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%). We will release all code and data.

2
Visual SKETCHPAD: Sketching as a Visual Chain of Thought for Multimodal Language Models
Yushi Hu*
Weijia Shi
Xingyu Fu²
Dan Roth
Mari Ostendorf
Luke Zettlemoyer'
Noah A. Smith
Ranjay Krishna
University of Washington Allen Institute for Al
University of Pennsylvania
Abstract
Humans draw to facilitate reasoning: we draw auxiliary lines when solving ge- ometry problems; we mark and circle when reasoning on maps; we use sketches to amplify our ideas and relieve our limited-capacino working memory. However, such actions are missing in current multimodal language models (LMs). Current thoought and tool-use paradigms only use text intermediate reasoning steps. this work, we introduce SKETCHPAD, a framework that gives multimodal LMs visual sketchpad and tools to draw the sketchpad. The LM conducts planning and reasoning according to the visual artifacts has drawn. Different from prior work, which
text-to-image models to enable LMs draw, SKETCHPAD enables LMs to draw with lines, boxes, marks, etc., which is closer to human sketching and better facilitates reasoning. SKETCHPAD vision models during the sketching process (e.g., draw bounding boxes with object detection models, draw masks with segmentation models), to further enhance visual perception and reasoning. We experiment on a wide range of math tasks (including geometry, functions, graph, chess) and complex visual reasoning tasks. SKETCH- PAD substantially improves performance on all tasks over strong base models with no sketching, yielding an average gain of 12.7% on math tasks, and 8.6% on vision tasks. GPT-4o with SKETCHPAD sets new state of the art on all tasks, including /"Bnnchh (80.3%), BLINK spatial reasoning (83.9%), and visual correspondence (80.8%).AM codes and data are in htttp:://visualsketchhe.
also specialist
gi sithub.... io/
Introduction
Sketching is a fundamental human activity, serving as a versatile tool for communication ideation [48], and problem-solving [47). Unlike written language, sketches have the advantage of conveying visuo-spatial ideas directly, for example by using spatial relations on paper to convey spatial relations or other more abstract relationships in the world. This may explain their ubiquity;
maps 40) and architectural plans [2] have been found incised in stone, etched on leather, impressed
in clay, and drawn on paper in diverse cultures scattered across the world [39] Sketches are so fundamental that we use them to teach school children how to solve geometry problems by drawing support lines, aid engineers conveying prototypes, support architects creating blueprints, and to allow scientists like us to convey scientific contributions (see Figure Tb.
As multimodal language models (LMs) 35 mature, we now expect them to solve tasks like the ones mentioned above, i.c., ones where people draw intermediate sketches simplify reasoning. Popular benchmarks now include questions
6 have begun to
Equal Contribution. Correspond to <Yushi Hu: yushihu@uw.edae <Weijia Shi: swj0419@uw.edd
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Geometry
Math Function
Prove the angles GABC to 188
SketchPad Pad GPT-4o
Function?
XK
SketchPad GPT-4o
Line DE parallel Bi
Ji =014 5.09
function
p GPT-4
4P
44 is conazuent to +2, and congruent to 23, <1, 24, and form straight angle 188" So 42, 42, 45 add up to 189".
for contrudietion,
158". intzoduce angle,
the
xxB
convex because Line between any two potntss Lse above the grapn.
(a)
(b)
Visual Search
Spatial Reasoning
Accordinng board, what Is the type the shop?
d
SketchPad GPT-4o
cooktes cookles?
Di
SketchPad GPT-4o
sliding. windav to
Use segnentation & nazking
Cooktes 2 and 5 seen to overlap. Need to analyze their depth
GPT-4o
see
GPT-4b
depth estination
The zonsed-in irsage shows the tast "cozepower YOGA". So 1t' yoga studio
Narner indtcate closer procinity, suggesting they are stacked on cooler-colose ones,
"cerenonial matcha, it's
No
(d)
Figure 1: SKETCHPAD equips GPT-4 with the ability to generate intermediate sketches reason over tasks. Given a visual input and query, such as proving the angles of a triangle equal 180%, SKETCHPAD enables the model to draw auxiliary lines which help solve the geometry problem. The examples are from [8 S1 44). For all these examples, without SKETCHPAD, GPT-4o fails to get the correct answer, while SKETCHPAD + GPT-4o achieves the correct solution.
about geometry (e.g., Geometry 3K 32 and complex math problems (e.g.. IsoBench (8). In these benchmarks, models are given images diagrams and asked questions requiring symbolic grounding and spatial understanding, where intermediate sketches like auxiliary lines can enhance reasoning. Even benchmarks in computer vision now have a similar flavor. Specialist vision models can be viewed as sketching on natural images. For example, object detection is plotting bounding boxes around objects; depth estimation is drawing colormaps according to depth. The recently proposed BLINK benchmark [9] would benefit significantly from such intermediate visual sketches. Similarly, V*Bench benchmark [5] could focus reasoning on image crops to find answers. Unfortunately, current LMs lack scaffold for using sketch-based reasoning when solving tasks.
In this paper, we introduce Visual SKETCHPAD: framework that provides multimodal LMS with the tools necessary to generate intermediate sketches to reason over tasks. Inspired by textual chain-of of-thought reasoning LMs
SKETCHPAD prompts the underlying visual LM to produce visual artifacts part of a chain of mixed textual, programmatic, and visual reasoning. For example, to prove that the angles of triangles sum up 180 degrees in Figure (a), SKETCHPAD enables agents to modify the diagram by introducing a new auxiliary line. This new line, along with new annotated angles, provides the critical information to solve the geometry task. Similarly, SKETCHPAD improves models' spatial reasoning for computer vision. To determine if there are cookies stacked on top of other cookies in the image (Figure [T)). the model first produces an intermediate depth estimate. By analyzing the depth estimate, which reveals cookies overlapping different depths, the model is able to correctly answer that the cookies are indeed stacked.
50 62
We demonstrate the effectiveness of visual SKETCHPAD across wide range of mathematics and computer vision tasks. For math, we tackle problems including (1) geometry [32, (2) mathematical functions, (3) graph algorithms, and (4) strategy games [8] For geometry questions, SKETCHPAD enables models to generate Matplotlib code with auxiliary lines and variables, given the diagram input and questions (Figure Tp). Notably, even when the input is solely language-based, such as mathematical functions, SKETCHPAD enables models to plot the functions and reason about their
properties, using only the mathematical function expression input (Figure Ip). These results

---

highlight the ability of SKETCHPAD to aid reasoning. even in tasks with purely language- bastnggeebbeddeee inputs. Across all four categories of mathematical tasks, SKETCHPAD consistently improves the baseline GPT-4o performance, yielding an average gain of 11.2% %. For computer vision, we tackle diverse tasks including (1) depth, (2) spatial reasoning. (3) jigsaw, (4) visual correspondence, (5) semantic correponnence, as well as questions from (6) the MMVP and (7) the V*Bench benchmarks 4 [1]] For this domain, SKETCHPAD enables models to generate segmentation masks, crop images, draw bounding boxes, zoom into image regions, overlay images, etc. Similar to math, SKETCHPAD shows consistent improvements across all seven types of computer vision tasks. For example, GPT-4o, augmented with SKETCHPAD, sees 14.3% improvement on V*Bench, 12.1%, 16, and 9.7% improvements on BLINK's ' depth and semantic correspondence tasks, setting a new state of the arts across all tasks. Finally, we analyze the effectiveness of SKETCHPAD by comparing the plans generated by model with human-created plans, showing that they are well-aligned and exhibit similar reasoning patterns. We hope SKETCHPAD opens up new research opportunities toward more capable and interpretable multimodal intelligence.
2 Related Work
SKETCHPAD generalizes recent work on multimodal tool-use and visual prompting. We also place our work within the larger sphere exploring LMs agents.
Visual programming and tool-use. With the advancement of LMs 35 42 45 13 researchers have demonstrated the possiblity of decomposing complex vision tasks into simpler substeps that can each be solved using vision tools [57, 60 18 [7). Among them, the most relevant to us are Visprog [14] and ViperGPT (38). They LMS to generate Python code, which sequencially invokes specialized vision tools. These methods share a common problem that the multimodal modules follow pre-defined plan outlined by the LM. By contrast, SKETCHPAD allows LMs to change their plan according to the intermediate visual artifacts, yielding better performance and robustness when solving complex multimodal tasks.
Visual prompting. Recent work shows that multimodal models can be augmented by visual prompts added to natural images
(37). For example, SoM (55) shows that adding labeled segmentation masks on images unleashes GPT-4V's visual grounding ability. Prior work also reports similar findings in 3D (25] and Robotics 34 SKETCHPAD is a generalized framework for all these methods, allowing LMs to decide what visual prompting to use as part of the multimodal reasoning process.
LMs as agents. Recent work has started to treat LMs as agents that can both reason and act 58 33
52 36 59 Researchers have applied this idea to software engineering 220 61 [5], robotics [34,,
agent that accepts multimodal inputs and outputs. One big difference is that SKETCHPAD can create visual
vision 29 57). and GUI navigation [54, 23 [33. SKETCHPAD can also be viewed
artifacts to facilitate reasoning, while prior LM agents only generate text during reasoning.
3 Visual SKETCHPAD
We introduce visual SKETCHPAD, general framework that enables multimodal LMs to draw sketches as intermediate reasoning steps and use these sketches to facilitate further reasoning. FFiuure 2 shows examples of how SKETCHPAD works. Given a multimodal query, SKETCHPAD agent generates sketching plan to address the query (Thought), and then synthesizes program create visual sketches (Action). By analyzing the resulting sketches (Observation), which serve as a visual representation of the reasoning process, the model generates final response to the query.
Our framework requires no finetuning or training. Multimodal LMs, out of the box, can be prompted to sketch using our framework. Our implementation is based on the AutoGen [52) framework. We give the overview our SKETCHPAD framework in $3.1 and delve deep into how it integrates sketching into the reasoning process in 3.2
3.1 Overview of SKETCHPAD
The SKETCHPAD agent solves tasks by engaging in an iterative interaction process with an envi- ronment. Given multimodal query g that includes both visual and textual components, the model generates a series of thoughts, actions, and observations to gather the information needed to answer the query. At each time step f, the model performs three key steps:
3

---

Geometry
Graph Connectivity
ven ,BCJ-105", 105" .FIC
is there sott between node addecency and the graph?
THOUGHT 1: need draw aux auxliary Line parallel to Line
THOUGHT 1:We can use the networko library plot the graph
ACTION
the using Python Code
ACTION 2 Python Ploting
OBSERVATION
coeecs, jusingets Dicreg))
polnnss
nu de.ddaac pos. sish, Labelantao
X.1 an wr.pin..p.. X.K) DV... V.O) "mot""
2.4 V.T. Slree bet
OBSERVATION 1
THOUGHT Given ,BCJ KKF 105", their supplementary angles, KEH .DDC 180 105" = 75". Given SD parallel to IX, it follows that _EX .KEH, 2XC <DCJ, both messuring 75%. Therefore,
THOUGHT 2: From the graph, saw there pash between node 0 and
FFI 150
(a)
(b)
Figure 2: Overview of SKETCHPAD. Given a multimodal query, the SKETCHPAD agent generates sketching plan to address the query (Thought), and then synthesizes program to create visual sketches (Action). By analyzing the resulting sketches (Observation), which serve as visual representation of the reasoning process, the model generates final response to the query.
Thought:
The model analyzes the current context C₁, which includes the query, previous thoughts, actions, and observations, to generate a thought plan pt for the next action. For example, given the query q- "find the LEIC" in Figure 2h. the model's thought plan pı is draw an auxiliary line XX parallel to BD serving as a visual sketch to help solve the problem.
Action:
Based the thought plan, the model executes action de, which manipulate both visual and textual content. In the geometry example, to realize the proposed thought of drawing the auxiliary line, the model generates Python code modify the original geometry diagram. The generated code is then compiled and executed.
Based on the action ag, SKETCHPAD'S environment returns new observation 01+11 such as a new diagram with the auxiliary line drawn in the geometry example. The multimodal context is then updated to C++11 (Ct, Pt, ae 0t+1).
Observation:
The multi-turn interaction process continues until time step T, when the model determines that has gathered enough information from the context cr to answer the query. At this point, it generates special Terminate action and provides the answer.
Different from prior work [58], where LMs primarily generate and manipulate text-based observations and actions, SKETCHPAD enables the model to work with multimodal observations o, and actions az, manipulating both visual and textual content. This allows the model to plan and reason with the visual sketches it has drawn, enhancing its problem-solving capabilities.
3.2 Sketching via Code Generation
The core component of SKETCHPAD is sketching, which enables the LM to generate visual sketches by synthesizing programs that call different specialist vision models Python plotting packages.
Program Generation. Similar to recent works like ViperGPT and VPD 14 38 18] SKETCHPAD enables LMs to sketch through code generation. The LM is provided, through prompt. with a detailed description of the available tools that can generate multimodal content (an example prompt and description can be found in $C) The prompt includes Python function signatures and docstrings [66 for these modules, but does not contain their full implementation. The LM generates Python code in code block, using the provided tools, which, when executed, generates new image and text outputs. A special display function allows the LM to visualize the sketch image in the next observation o1+11
Modules for sketching. SKETCHPAD uses various tools to facilitate the sketching process, depending on the task at hand. For mathematical tasks, SKETCHPAD uses common Python packages like

---

matplotlib and networkx for plotting (see $ For vision tasks, the LM leverages specialist vision models during the sketching process. These models include detection tools that draw bounding boxes on the image, as well as segmentation and marking tools (inspired by SoM [55] that draw colorful masks on the image and use numbers label segments. We find these specialists possess useful perception skills for visual reasoning tasks, and SKETCHPAD is an effective way to combine them into a multimodal LM (see $5.1).
4 Sketching to Solve Math Problems
In this section, we experiment with SKETCHPAD on four complex mathematical tasks : (1) geometry, (2) mathematical functions, (3) graph algorithms, and (4) game strategies. We demonstrate that incorporating sketching capabilities into LMs significantly improves their performance on these mathematical problems, setting new ofte-oof-thhart results ($4.1.
Details of our evaluation tasks and the tools employed for visual reasoning are as follows:
Geometry Problems.
Drawing auxiliary lines in geometry diagrams is often helpful for problem- solving. For example, in [Figure 2 (a), when asked find ZEIC, the LM plans to draw auxiliary line IX parallel to BD, allowing it to use the properties of parallel lines to determine LEIC. To evaluate the effectiveness of SKETCHPAD, we use the problems from the Geometry3K dataset (32]
To realize the line drawing process, SKETCHPPD takes a geometry diagram and its corresponding matplotlib code input. The model then proposes and modifies the code to generate auxiliary lines, and executes the modified code to visualize the updated diagram with the added lines.
Mathematical functions. Understanding mathematical functions is crucial for various applications in science, engineering, and economics. We focus on two tasks related mathematical functions from the IsoBench datasets g
Classifying parity aims to determine whether a function is even, odd, or neither. Even functions satisfy f(-x) f(x) for all x, while odd functions satisfy f(- fx)).
Identifying converity/concavity aims determine whether function is convex concave.
Existing LMs can only analyze functions and attempt to prove their properties analytically.
However, SKETCHPAD enables them to visually sketch functions to solve problems more efficiently. For instance, to determine the convexity of the function in Figure 1p, SKETCHPAD allows the model to plot the function using matplotlib, and visually inspect its overall shape.
Graph algorithms. Many real-world problems, such as those related to computer networks and transportation systems, can be formulated as graph problems. We evaluate SKETCHPAD three graph problems from IsoBench [8]:
Graph connectivity determines whether there exists a path between two vertices graph.
Maximum flow aims to find the maximum amount of flow that can sent through a network from source vertex sink vertex, subject to capacity constraints the edges.
Graph isomorphism tests whether two graphs are structurally equivalent.
Given an adjacency matrix of a graph like in Figure 2))).. SKETCHPAD can draw the actual graph structure, using using Python's 's networkx library, enabling direct visual reasoning about graph properties and relationships.
Game strategies. Chess games can be represented in various formats, including visual board states and textual move notations. Given only the textual move notations, SKETCHPAD can draw the visual representations of the chess board analyze positions and formulate strategies. We evaluate the performance of SKETCHPAD on the winnder identification task from the Isoßench datasets [8] that aims find the outcome of chess game (win for White, win for Black, draw) based the final board state. To create the graphical board, SKETCHPAD uses Python's chess library to draw the board from the Sorsyth-Edwards Notation (FEN) of chess.
For humans, the analytical approach is the correct way tackle these math tasks. However, we observe that LMs are not good at analytical reasoning in math. They make errors when deducing f(=x) and derivatives.
5

---

Geometry
Graph
Math
Game
Model
Geometry
Maxflow
Isomorphism
Connectivity
Convexity
Parity
Winner ID
Prior LLMs without visual inputs
Gemini-Pro
15.6
47.7
50.0
87.9
48.2
8.1
Claude 3 OPUS
56.3
50.0
82.0
93.0
77.6
74.4
Mixtral 8x7B
19
8.6
50.0
62.5
69.1
41.7
LLaMA-2-70B
46
18.0
50.0
50.0
74.2
33.3
12.4
Latest multimodal
Visual Sketchpad
Gurbo
Turbo
37,5
32.8
62.5
66.0
57.0
80.5
50.4
Sketchpad
97.6
97.6
77.3
71.5
+8.3
+64.0
+35.1
+31.6
+20.3
-9.0
+13.8
GPT-4o
62.5
25.0
50.8
96.1
87.2
84.4
61.1
Sketchpad
66.7
66.3
65.3
98.4
94.9
94.7
64.6
+4.2
+41.3
+14.5
42.3
47.7
+10.3
43.5
Table 1: Accuracy scores geometry problems, graph algorithms, mathematical functions, and game. SKETCHPAD yields large performance gains on most tasks and outperform all baselines.
4.1 Results
We evaluate the performance of SKETCHPAD on multimodal LMs with API access, including 100-4-tuubo-22224
We compare these results to baselines without the Visual Sketchpad and other notable closed-source models, such Claude 3 and Gemini- Pro, as well as open-source models like Mistral 19 and LLaMA-2 70B 46
and gpt-40-2024-05-11
Main results. As shown in Table SKETCHPAD consistently improves base model performance all tasks, with an average improvement of 11.2% for GPT-4o and 23.4% for GPT-4 Turbo. In particular, we observe large gains on graph algorithms such as maximum flow and connectivity. For instance, GP with SKETCHPAD achieves accuracy of 66.3% the maximum flow problem, improving over the base model by 41.3%. Similarly, SKETCHPAD substantially improves the performance on mathematical functions, with GPT-4 Turbo achieving over 90% accuracy and GPT-4o over 88% accuracy on convexity and parity classification tasks. Furthermore, we observe gains (3% 10%) on game strategies, demonstrating that drawn game boards drawn can improve reasoning about game strategies. Overall, these results highlight the effectiveness of SKETCHPAD in enhancing the reasoning capabilities of multimodal language models across diverse domains.
5 Sketching to Solve Computer Vision Tasks
In this section, we experiment with SKETCHPAD on complex visual reasoning tasks. Recent work (BLINK) [9] finds that many visual perception abilities are still missing from cxisting multimodal LMs-even though many computer vision specialist models possess such abilities. Also, SoM (55] shows that drawing segmentation masks images unleashes the strong visual grounding ability of GPT-4V. We generalize these ideas with SKETCHPAD, allowing LMs to use specialist vision models sketch. Details of these modules in $5.1 SKETCHIAD enhances multimodal LMs' visual reasoning abilities and establishes new SOTAS on all 7 tasks ($5.2).
Tasks. We experiment with wide range of complex visual reasoning tasks: (1) V Bench [51] This benchmark contains questions about small items in an image. (2) MMVP benchmark from Eyes Wide Shut [44]. This benchmark contains visual questions specially designed to reveal the visual shortcomings of based multimodal LMs. (3) BLINK [9] This benchmark contains visual perception tasks that easy for humans, but post significant challenge for multimodal LMs. Specifically, we experiment with relative depth, spatial reasoning, jigsaw puzzle, visual correspondence, and semantic correspondence tasks. More details of each task are in $
5.1 Vision Specialists as Sketching Tools in SKETCHPAD
LMs the following modules to sketch and manipulate images. We wrap these modules into Python functions that the LMs can call. Refer to for the function definitions.
Detection. This module takes an image and a simple text query (e.g., "cat") as input. We run the Grounding-DIne DINO 30 open-vocabulary objection detection model and plot the detected bounding boxes (together with number label) on the image. It also returns the bounding box coordinates.

---

Relative depth
Visual Search
Which point closer the camera, or 0?
Tel player who is shooiing
THOUGHT 1:1 can use depth estimation tool that provides depth with colore ind veliating distances.
THOUGHT 1: need to detect the players in the image and room in player who shooting the number clearly
1: depth cotmmtion
objject detection
21 cl JF4P
testhcinege)
Fon fey Temae deteet detecttie
Smape)
THOUGHT depth mep colorss purple Narthed yellow [cooee)) fnawdd to overlay the original image with the mapa
image, Player2 sooms to shooting. zo0m on this player see the number on their jersey cleary
ACTION 2:
S
Call cverlyy
ACTION 2:
S Call zoom-in
mweec seorlay, sagee smag
Feoeeol mome O4, Temer by thone 1. shooter)
THOUGHT
that point yellow regian, the camera compared to point A, which the purple region
zoomed image, clearly see number of the player who is shooing is
THOUGHT
(a)
(h)
Figure Examples of SKETCHPAD applied vision tasks. The figure shows actual outputs generated by SKETCHPAD. By contrast, the baseline GPT-4o model cannot answer these questions correctly. Note that for demonstration purposes, the "A" and "B" marks in (a) are different from the actual images in the experiments.
Segmentation. This module takes an image as input and returns image with colorful segmentation masks on it. Each mask also has a number label. We follow the implementation of SoM [55]. The underlying segmentation models are Segment [22] and Cemantic-SAM. 224]
Depth estimation. This module takes an image as input and returns a depth map. The underlying model is DepthAnything [56].
Visual search via sliding window. This module mimics how humans search for small items on an image. It takes a text query as input and runs a sliding window over the image. The window size is 1/3 of the image size, and the step size is 2/9 of the image size (so image will have 4 16 windows). It returns a sequence of image patches in which the query is detected.
Other image manipulation modules. Other modules include (1) zoom-in and crop, which takes an image and a bounding box as input and returns the image patch inside the box; (2) Overlay images, which takes two images and alpha values input, and returns the overlayed image.
5.2 Results
We experiment with the same multimodal LMs as in $ on complex visual reasoning tasks. We compare the performance with and without SKETCHPAD, well other notable multimodal LMs, including Gemini 442] Claude (2). and the open-source LLaVA 1.5 266, LLaVA-NeXT [27].
Main results. Table shows the performance of our SKETCHPAD and baselines. SKETCHPAD consistently improves base model performance across all tasks, GPT-4o with SKETCHPAD sets the new state tifte-of-the of-the-art results on all tasks. SKETCHPAD is particularly effective on Bench, yielding 18.5% accuracy improvement for GPT-4 Turbo and 14.3% improvement for GPT-4o, surpassing the previous state of the art SEAL 51 which used visual search model specifically trained for this task. On BLINK tasks, SKETCHPAD on average yields 6.6% absolute accuracy gain for GPT-4 Turbo and 9.0% gain for GPT-4o. Interestingly, despite the fact that all modules in SKETCHPAD work on a single image, the LMs also get substantial improvement on multi-image tasks, including jigsaw puzzles, visual correspondence, and semantic correspondence. Finally, GPT-4o, the LM with stronger multimodal ability than GPT-4 Turbo, benefits more from SKETCHPAD. For example, on the relative depth task, GPT-4o gets 12. % accuracy improvement, while GPT-4 Turbo only gets 2.4%, showing that 4PT-4o better at understanding the depth map SKETCHPAD generated. Overall, experiments show that SKETCHPAD is effective way improve multimodal LMs' performance on visual reasoning tasks.
How many times is each vision specialist used? We count the number of times each vision specialist is used in each task, as shown in Figure 4 Here we choose the four tasks that achieve the largest improvement: J"Bench, relative depth, spatial reasoning, and semantic correspondence.
7

---

Model
V Bench
MMVP
Depth
Spatial
Jigsaw
Vis. Corr.
Sem. Corr.
Prior multimodal LLMs
LLaVA-1.5-7B 26
48.7
52.4
61.5
11.3
25.6
23.0
LLaVA-1.5-13B
26
24.7
53.2
67.8
29.1
32.4
LLaVA-NeXT-34B
27
67.7
74.8
54.7
30.8
23.7
Claude OPUS
47.6
58.0
32.7
36.6
25.2
Gemini-Pro 42
48.2
40.7
40.3
74.8
57.3
42.4
26.6
GPP--V-preview
55,0
38.7
59.7
72.7
70.0
33.7
28.8
Previous state of the art
75.4 [S]
49.3 1T0
67.7 10
76.2 [
70.0 35
42.4 42
33.1 [9]
Latest multimodal LLMs
Visual Sketchpad
GPT-4 Turbo
52.5
71.0
66.1
68.5
64.7
48.8
30.9
+ Sketchpad
71.0
73.3
68.5
80.4
68.5
52.3
42.4
+18.5
+2.4
+3.5
GPT-4o + Sketchpad
66.0
85.3
71.8
72.0
64.0
73.3
48.6
80.3
86.3
83.9
81.1
70.7
80.8
58.3
+14.3
+1.0
+12.1
+9.1
+6.7
+7.5
+9.7
Table Accuracy on complex visual reasoning tasks. SKETCHPAD enhances both GPT-4 Turbo and GPT-4o performance, establishing new SOTA performance levels on all the tasks.
We observe that (1) the use of vision specialist is task-dependent, and the two LMs analyzed utilize similar tools. For example, for V*, which needs to locate small objects, the LMs mainly use detection, sliding window search, and zoom-in, similar how people would search. For the relative depth task, both models rely on depth estimation. For spatial reasoning, the LMs use detection and segmentation facilitate visual reasoning. (2) GPT-4o likes to use more tools. GPT-4o uses the vision specialists more often than GPT-4 Turbo. Also, the two LMs behave differently for the semantic correspondence tasks, GPT-4o uses the segmentation module for 40% of the task instances, while GPT-4 Turbo uses the detection module for less than 20% of times, and rarely uses the segmentation module. This difference may explain the performance gap between the two LMs (58.3% v.s. 42.4%) on this task.
4o Tbal
Detaction
Model
MMVP
Depth
Spatial
GPT-4 Turbo
$2.5
71.0
68.5
SoM
42.0
60.7
58.9
78.3
SoM + orig.
51.3
74.3
66.9
79.7
Visprog
33.2
16.3
67.8
53.8
Vole
Sketchpad
71.0
73.3
68.5
80.4
GP14 Turbo looltise Frequecyy
GPT-4o
66.0
85.3
71.8
72.0
Segloeration
SoM
49.0
70.7
62.9
83.2
Zeemis Siding siidon
SoM + orig.
68.1
84.0
75.0
82.5
Visprog 32.4
17.3
46.8
37.8
Sketchpad 80.3
86.3
83.9
81.1
Table 3:
Comparison with other augmenta- tion frameworks for multimodal LMs on single- image tasks. For fair comparison, we modify the original Visprog 14] framework by replac- ing the LM and VQA components with the cor- responding GPT-4 model.
Figure 4: Percentage of times GPT-4o and GPT- 4 Turbo use a visual module in SKETCHPAD when solving V Bench, relative depth, spatial reasoning, and semantic correspondence tasks.
Comparison with visual prompting and tool-use frameworks. In Table B we compare SKETCH- PAD with the visual prompting framework SoM (55) and the LLM tool-use framework Visprog [14). Details of these methods can be found in $2 For fair comparison, tations: (1) we find that prompting LMs with SoM images can hurt performance, likely because the visual prompts confuse the model. To make stronger baseline, we prompt the LM with both the original image and the SoM image (full prompt in $C). which we refer as "SoM + orig." (2) replace the LM and VQA modules in Visprog with the corresponding GPT-4 model. (3) Since baseline methods are developed on single-image tasks, we compare SKETCHPAD such tasks. From Ta we can see that SKETCHPAD is the only framework that yields consistent improvement on all tasks. SoM can boost spatial reasoning ability, the authors reported. However, it can hurt the
make the following adap-

---

performance on other tasks, even in the "SoM + orig." setting. Visprog performs worse than the base LM on all the tasks. As prior work [8] suggests, one possible reason that the vision modules themselves have errors, and the error propagates when the modules are composed by a program.
6 Analysis and Discussion
Why does SKETCHPAD work? First, vision is versatile and informational interface that complements language. Dense information like depth and segmentation cannot be described easily through language 19. In broader perspective, humans have developed many visualization techniques that are direct, efficient, and informational. SKETCHPAD provides LMs the opportunity to use them. Second, in SKETCHPAD, multimodal LMs can plan and reason based on the intermediate visual artifacts they created. In contrast, in prior modular vision work [4 38 55), multimodal modules follow predefined plan by either humans or code. SKETCHPAD is much more flexible and robust to enrors. For example, suppose object detection makes error. The LM can (in principle) find the error by viewing the bounding boxes, and change its following plans, but prior methods cannot. Third, discussed next, the plans of multimodal LMs are similar to human plans, and therefore likely benefit from the fact that the underlying LMs have seen data with similar reasoning patterns.
Do LMs have the same plans as humans? We conduct human study on all geometry problems and 10 problems on each vision task. On geometry. humans draw the same auxiliary line as GPT-4o 80% of the time. On vision, we show 2 human subjects the full plan of GPT-4o, which they rate is valid in 92.8% of instances. Most errors are caused by failures the vision specialists (e.g.. fail to detect object) and mistakes in simple visual question answering, rather than planning.
Model
Geometry
Maxflow
Convexity
Winner ID
LLaVA- NeX--N3T-11
11.1
7.8
50.39
5.8
+ oracle Sketchpad
22.2
10.2
50.0
36.7
LLaVA-NeXT-34B
26.1
0.8
81.6
49.0
oracle Sketchpad
28.3
14.1
87.1
49.4
Table 4: Open-source LLaVA models' performance on math tasks. The oracle Sketchpad uses the visual artifact generated in the last action of GPT-4o SKETCHPAD as inputs.
Experiments on open-source models. Can sketches like diagrams, plots, and auxiliary lines facilitate existing open-source multimodal LMs? To answer this question, we conduct the experiments in Table We use the visual artifacts generated in the last action of GPT-4o + SKETCHPAD experiment the image input for open-source LLaVA- models (27]. We see that this oracle SKETCHPAD brings consistent improvement to math tasks and boosts mathematical reasoning.
Conclusion
We present Visual SKETCHPAD, a framework that provides multimodal LMs with the tools necessary to generate intermediate sketches to reason over tasks. For complex mathematical reasoning tasks, SKETCHPAD yields large performance gains, by visualizing auxiliary lines, math functions, graphs, and games during reasoning. For visual reasoning tasks, we add vision specialists to SKETCHPAD. The LM can call these specialists during reasoning, observing the visualization of these specialists' predictions (e.g., bounding boxes from the object detection model; masks from the segmentation model), and then conduct further planning and reasoning. Experiments show that SKETCHPAD enhances the LMs' performance across all tasks, and sets new state-of-the-art results. Ultimately, SKETCHPAD represents a step toward endowing LMs with more human-like multimodal intelligence, leveraging the complementary strengths of language and vision to tackle increasingly complex reasoning challenges.
Limitations and future directions. First, SKETCHPAD requires more computing resources than directly outputting language tokens. We discuss more about computing costs in E Second, this work focuses on existing off-the-shelf LMs. Future work may explore the training side of SKETCHPAD. For example, recent multimodal models like Unified-1O 2 BI and Chameleon 41] are natively multimodal and can output both text and images. SKETCHPAD may emerge as a new paradigm for instruction tuning these models. Finally, SKETCHPAD can be applied in more areas. For example, for robotics, we can apply SKETCHPAD to search for small things in a crowded space, highlight the object of interest, and zoom the camera for better view or depth estimation to help navigation.

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances Neural Information Processing Systems, 35:23716-23733 2022.
[2] Anthropic. Introducing the next generation of claude. http:///wwwwaabrpiccccco/////// claude- 3-mmmil March 2024.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[5] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang. Ceslee Montgomery, Paulina Pictrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up multilingual vision and language model, 2023.
[6] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Sealing up vision foundation models and aligning for generic lingul-iin tasks. arXiv preprint arXiv:2312. 14238, 2023.
[7] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[8] Deqing Fu", Ghazal Khalighinejad", Ollie Liu", Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomorphic representations, 2024.
[9] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint 2rX4v:2404.12390 2024.
[10] Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv: arXiv:2402.05933, 2024.
[11] Vinod Goel. Sketches of thought. MIT press, 1995.
[12] Gabriela Goldschmidr. The dialectics of sketching. Creativity research journal, 4(2):123-143,
1991.
[13] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.0088. 2024.
[14] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953-14962 2023.
[15] Sirui Hong, Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2023.
10

---

[16] Cheng- Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arXiv preprint 1rXiv:2308.006755 2023.
[17] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.0969) 2022.
[18] Yushi Hu, Otilia Streteu, Ch Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. arXiv preprint arXiv:23 (oXix:23322000: 12.03052, 2023.
[19] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[20] Carlos Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
[21] Apoorv Khandelwal, Ellie Pavlick, and Chen Sun. Analyzing modular approaches for visual question decomposition. arXiv preprint trXiv:2311.06411 2023.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yes Yen Lo, et al. Segment anything. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[23] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXiv:2401.138 2024.
[24] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767 2023.
[25] Dingning Liu, Xiaomeng Dong, Renrui Zhang, Xu Luo, Peng Gao, Xiaoshui Huang, Yongshun Gong, and Zhihui Wang. 3daxiesprompts: Unleashing the 3d spatial task capabilities of gpt-4v. arXiv preprint arXiv:2312.0977 2023.
(26) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[27] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[28) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[29] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning use tools for creating multimodal agents. arXiv preprint arXiv:231 2tth:23311.0437, 2023.
[30] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded me-training for open-set object detection. arXiv preprint arXiv:2303.cm 05499, 2023.
[31] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023.
[32] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chur Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL.), 2021.
11

---

[33] Zixian Ma, Weikai Huang. Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m's: A benchmark evaluate tool-use for multi-step multi-modal tasks. In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024.
[34] Soroush Nasiriany, Fei Xia, Wenh Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang- Wei Edward Lee, Kuang- Lee. Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. 2024.
[35] OpenAI. Gpt-4 technical report, 2023.
[36] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: -eugeeee-- black-box language models. In NAACL, 2024.
[37] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11987-11997, 2023.
[38] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. Proceedings of IEEE International Conference Computer Vision (ICCV), 2023.
[39] Holly A Taylor and Barbara Tversky. Descriptions and depictions of environments. Memory & cognition, 20:483-496, 1992.
[40] Holly A Taylor and Barbara Tversky. Spatial mental models derived from survey and route descriptions. Journal of Memory and language, 31(2):261-22) -222, 1992.
[41] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. 2024.
[42] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 1rxhv:231..//800 2023.
[43] InternIM Team. Internim: A multilingual language model with progressively enhanced capabilities.
http://tttuub..mmmmmteeeee Cit.om/mnnnenn/////mmm Iww.LM/MIIeettMMM. 2023.
[44] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms, arXiv preprint arXiv:2401.06209, 2024.
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cymthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[46]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut
12

---

Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iiyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[47] Barbara Tversky and Masaki Suwa. Thinking with sketches. 2009.
[48] Barbara Tversky, Masaki Suwa, Mancesh Agrawala, Julie Heiser, Chris Stolte, Pat Hanrahan, Doantam Phan, Jeff Klingner, Marie-Paule Daniel, Paul Lee, et al. Sketches for design and design of sketches. Human Behaviour in Design: Individuals, Teams, Tools, pages 79-86, 2003.
[49] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03000 2023.
[50] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[51] Penghao Wu and Saining Xie. V*: Guided visual search as core mechanism in multimodal Ilms. ArXiv, abs/2312.14133, 2023.
[52] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023.
[53] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972. 2024.
[54] An Yan, Zhengyuan Yang. Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang. Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint 13Xx::331.00566 2023.
[55] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023.
[56] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scall unlabeled data. In CVPR, 2024.
[57] Zhengyuan Yang*, Linjie Li*, Jianfeng Wang*, Kevin Lin*, Ehsan Azarnasab*, Faisal Ahmed* Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. 2023.
[58] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint urXiv:2210.03622. 2022.
[59] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. language modeling. In International Conference on Machine Learning, pages 39755-39769.
multimodal
PMLR, 2023.
[60] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0m) 00598, 2022.
13

---

[61] Jieyu Zhang. Ranjay Krishna, Ahmed H Awadallah, and Chi Wang. Ecoassistant: Using llm assistant more affordably and accurately. arXiv preprint arXiv:2310.03046, 2023.
[62] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00993, 2023.
14

---

Appendices
Acknowledgement
16
B More Qualitative Examples
16
Prompts
17
D Dataset Statistics
26
Costs
26
Impact Statement
26
15

---

A Acknowledgement
This work in part supported by gifts from Allen Institute for Artificial Intelligence, Meta, Apple, and Amazon Science. We also greatly appreciate the help from Zixian Ma, Wei-Chiu Ma, Benlin Liu, Jieyu Zhang, UW NLP, AllenNLP, A12 PRIOR, and UW RAIVN for their help.
B More Qualitative Examples
Here provide more details of how GPT-4o and SKETCHPAD solve complex mathematical and visual reasoning tasks.
Math Function
Correspondonce
Is fll even, odd, neither? 100 2- 12e" + 11.16
?
point is correspanding to the refermnce point?
THOUGHT functioss are symmatnc the yours, gymmetric about the orign Uneed piot the Lictton
THOUGHT canespording paint reference point mage, the segine raaiinn and ma sing tool undentand objects relofinnsh ps
ACTION 2.
Call Fathon Poting too!
ACTION
Col bug mentaaion and malking
Exsest seppin anaut
OBSERVATION
THOUGHT symmetrc about angn, inicating
THOUGHT 2:the reference
toeeficlly the second need find
The regmentaionn marking corsepencing paint rght lubeled
that
Figure 5: Qualitative examples of how GPT-4o and SKETCHPAD solve (a) complex mathematical and (b) visual reasoning tasks.
16

---

C
Prompts
Prompts for computer vision tasks. Here we provide the running log of GPT-4o SKETCHPAD tackling a visual search problem from V"Bench [5]]. We use the same prompt template for all computer vision problems in this work. For visualization purposes, we present the prompts that contain codes differently. In our implementation, they are all text inputs to the LLM.
SYSTEM PROMPT You are helpful multimodal Al assistant. [MORE INSTRUCTIONS ...]
For each turn, you should first do a "THOUGHT", based on the images and text you see. If you think you get the answer to the intial user request, you can reply with "ANSWER: <your answer>" and ends with "ERMINATE".
Initial Prompt Request
help you. are python codes. They are tools py and will imponed you. The images has their own courdimate system. The upper left corner of the image is the origin (0. 0), All coordinates are normlizedd i. the range [0, II.
are some tools
All bounding boses
of the box, the vertical coordinaee that comer. hl. which python list is the horizental courdinate of the upper-left box
the format
the height.
Notice that you, as an Al ussistant not good at locating things and describe them with coardinate. can use tools to genenut bounding boxes
You also good answering questions about small vicual details image. You tools tools .py:
image the details.
are
python
class Ansotatedimage:
class represent annotated image. contains the innotated image the original
10
def -iint ( setf, unsotated Image: Image Image, cniginalimage: Image..ingeeNiae. self amotatedd_magee annotaee image self original_jmage ceiginal_inage
11
12
13
14
def detection (image, objects);
15
""Ceject detection using Grounding DINO modal It retunas the annotated image and the bounding botes of the derected objects. simple
16
(e.g- "bus", "red, cur"). Cannot hard the model wall break. some objects.. Also, notice that the bounding bas label might be out of the image bousdar..
17
detestor is not perfect. may wrongly detect objects
18
19
You abould use the outpur reference, not ground truth.
20
When anvwering questions the imago, you should double-check detected objects.
22
Args:
23
image Image Image): the input image
24
objects (List]
"number". objocts desect. Each object should simple simple Should be abstract
concepes like
25
26
Remuns::
oulput imago (Annotseellenuge) original image, annotated with bounding hexes, Fach laboled with detected object, and
28
processed boxes (List): listthe bounding boxes of the detected objects
29
30
Example:
31
image Image opent "sample sin.jpg")
12.
output image, boxes derection (image, ["bus"])
33
display (output_inan www.ammmmattteeeeee...eee onnotated image)
34
(boxes) [10.24, 0.21, 0.4,
0.3]]
35
you to dhuble-chock detected objoats. Soms objocts may he missed wrongly detected.
36
37
38
diding window detection (image, objects)
39
Use whene you scarching objects image, objects are detected object détection model.
40
In that case, the most common resson that the object is too small such that both the vision-longuag model and the object detection model
fall deteet
fonction
object sliding
42
With help of the detection model, it tries ditect the object in the zomedd-n putches.
43
The function retuns a Hat of annotated images that may contain at less one of the objects, annotated with bounding hoxes.
44
Ii also returns list of list bounding boxes of the detected objects.
45
46
Args
47
image Image Image?: the input image
48
objocts (Eis (tr list of objocts to detect Each object should be simple noun simple phmss. Should not he hard or abstract "number".
concepes like
49
50
Retums::
51
possible patches (Liss [Annotatedimmgel:: possible boxes [List [Flost IID:s imsaee in possible pulches cocedinates tee W.r.t. ench zoomed-in imago. The onder the boxes, as the onter the images possille patches.
of annotated noomed-in that may contain the object, unnolated with bounding boxes. bounding boses detected objects
52
53
54
55
Example:
image Image opent 'sample ipg")
17

---

possible_ patches. possible_boxee objjctt Joomtimige, I"burd", "sign
58
i. putch enumeratel possible,
59
(f (1))
60
display panch imaage)
62
8 print bounding boxes desecsed objects fiss patch
63
print possibic__ boxes (O)) 110.24, 0.21, U.S. 0.41, 10.5, 0.3, 0.2, 0.311
64
66
sggment and marktinuge. anno modellist M'Mak", "Mar']]:
67
Use segmentation model to segmest the image, and add colortal musks on the segmented objects. Each segient is also laheled with a number. The annotited image is returned along with the bounding boxes of the segmeensdd objects
68
This nod may help you better
NOT use this to search detect object. rolaiceship
objoets, which be useful for spatial ressoning
70
likely object small segmestaiton Segrentttion and marking can also be helptul for 3D and video reasoning. . For exangle, helping you to see more clearly and analyzes the relationdlip between different frimes of video.
not help.
71
12
73
Args:
74
image (PIT.. Image Imagey: the ingut image anno mode (list, optional): ); What annocation added the inpur inagee Mask is the colorttal masks, And mark is the number Inhels. ['Mask.. "Mark"].
75
Defaults
76
77
Retuni::
78
output. (Annotaielliruge): the original image annotated with colorful masks and number labels, Each mask is labeled with number. The
number label
79
bhones [List): bomoding boxes masks The
same the
labels.
80
81
Example:
User request:
find seat window, should
83
Code:
84
"python
85
image Image 'sample iigg ing')
output imago, Nhoxs segment marktimige)
87
display (outpur_inage annototed, image)
88
89
Model reply: You can sit on the chuir numbered 5, which close the window
me the bounding python
91
Code:
92
93
[bboses[[]
[0.24, 0.21, 0.3, 0:4]
95
Model reply: The bounding
chair numbered is [0.34, 0.21, 0.3, 04).
96
97
depthfimage::
99
Depth estimation using DepthAnything model. It returns the depth mup of the input image.
100
A colornup used represent the depth. It. uses Infemo colornago The closer the object, the wammrr the color.
This tool may help you better about the spatial relationship like which object is closer the cumera.
Args
104
image (PIL. Image': the input image
105
Retums:
107
outputimage (PIL Image Image): the depth map the input image
108
Example:
image Image opent 'samplee jiiiiiig.g. ipg')
sulput image (epph(image)
112
display (ougput_mmage)
113
115
def zoom, in image, by Lbbuxtinage, box, pudding=0.005;
116
A simple wrapper function to crop the image bused on the bounding bux.
when
about visual desails bounding annotated by the detection tool, you would
object using function.
118
119
Args:
120
(PIL, Image Image's input image
D): bounding
[x, y. h]
122
padding ( Boot optional ): The padding for the image crop, outside the bounding box. Defontts to 0.1. The zoom factor cannot be too
small. Minimum 0.05
Returss:
125
inu Image Image': the cropped image
126
Example:
image Image. opent 'sample jing.ig"")
129
mmntttted img. boxes detection timage. "bus")
130
croppedjing Jmg zoom in iaciiinnnnaneee image, by bbox/image, boxesf0], padding=0.015)
display (crapped img))
133
134
def overlay iwwgesshhcckkrnnneee img, overtay jong alpha -0.3, bounding nox-di, 1, ID:
Overlay an image another image with transparency.
137
This is particularly useful visalizing heatmop while preserving some info from the origital image.
138
For example, you can overlay segmeated image a heatmap to bettrr understand the spatial relationship betwees objocts.
also help sezing the labels, circkes the original image may visible the hestmap.
141
Args:
142
background iig. pil (PII. Image Image):: The brokground image in PII, format. overlay Imag. The image to oweray
18

---

alpha (Boat Franeprency of image. boundingg (Liuj Dr: bounding box ovartay image: bockgrundd image. Defonits to (0, u. I. LJ.
[K, Y: BJ. The coondiatess
146
PIL.Ionage.Inge. resuiting, image overley, fonsat.
149
150
Example:
image. original .jpe")
depth mag = depth(inage)
153
overlayol, image overtay. jdngess(dephh, map, image, alphi-0.3)
154
display (overlayed_ imsge))
157
GOAL Based on the above tools. want you to reason about how to solve the USER REQUEST and generate the actions step by step (each action
python jopyter notebook code block) to solve the roques
may ability not perfect, you
decisions based visual oupputs
bloeks.
assist you reasoning images.
160
161 a ssx co-comtext examples hiere]
REQUEET
the rod balkon alowe or below the white ballown? Options: (A) The rod ballown helow the white balloon. (R) The rod halloen above the while balloon.
USER IMAGE stured in image. I PIL. image.
Now please penerate only THOUGET O und ACTION 0 RESULT. no action neoded, alsu reply with ANSWER: syour answerd and ends with TERMINATE the RESUL.T:
RESULT
Thought + Action 0
THOUGHT O. 1o determine the relative positions of the red and white balloans, need to detect the ballooas the Image. will use the detection tool to find the balloces.
ACTION
python
image. I balloon desection, image, balloon, boxes detection (imago_).. ('red balloon "white ballocn D)
display timage. balloon, detection anoorated_jaaag
19

---

Observation 0
Execution success. The output as follows.
Please generate the next THOUGHT and ACTION, you can get the answer, please also reply with ANSWER: <your answer> and ends with TERMINATE.
Thought + Action 1
THOUGHT detcetion tool did balloass image. white balloons
will slidingg window detection scarch
image.
ACTION I:
python
slidiing for i. patch in enumeraiel possible_guiches ): print Pa"Ph (i+ll:")
possible patches. possibble
detection (image 1, ["rod balloon", "white balloon's ID)
display (potch. annotited imago)
Observation 1
Execution success. The output is as follows. Patch 1:
SXPTEP KMAT
beloon
20

---

Patch 2:
SPPEP KAHD
Patch
Patch 4:
looe
Patch
Please generate the next THOUGHT and ACTION, you can get the answer, please also reply with ANSWER: <your answer> and ends with TERMINATE.
Thought + Action 2
THOUGHT 2: From Patch 4 and 5, I can see both the red and white balloons. The red balloon is clearly below the white balloon in these patches.
ACTION 2: No action needed.
ANSWER: The red balloon is below the white balloon. So the answer (A). TERMINATE
21

---

Prompts for math tasks. The prompts are similar to the vision task for math tasks, except we remove the computer vision specialists, and add example codes for math plotting. Besides, the user query is different. For each task, the user query part of the prompt is as follows.
PROMPT
You are given a real-valued, scalar function fz).
YOUR TASK is to determine whether f()) is even function, an odd function, neither. Definition of an odd function: function such that
f(-x) = -f(x)=
where the sign is reversed but the absolute value remains the if the sign of the independent variable is reversed. function is neither even nor odd it does not satisfy either condition. Here is the expression of (x)):
-2x3
f(z) 2xd 4x + 12x 4x 11.16
Respond with 'even', 'odd', 'neither' first on whether the function f(x) even, odd, or neither, based on the definitions and your observation of the function. You can generate matplotlib code to visualize the function. If you can get the answer, please reply with ANSWER: <your answer>, extract the final answer in PINALANSWA Answer:
<final answer> and ends with TERMINATE in the RESULT.
Figure 6: Prompt for the Math Parity task. We follow similar prompt format g except prompting the models to write the code to generate images.
PROMPT
You are given real-valued, scalar function fz). YOUR TASK determine whether f(x) is an convex function or concave function. Definition convex function: A function such that
Yz, y, 0 St 1, f(tz (1- t)y) (f(x) t)f
Definition of a concave function: A function such that
Yx, 1.,0sts11 f(t t)y) tf (z) + (1 t)f(v)
Here is the expression of f(x):
f(x) 7.57 Abs(r)
Respond with 'convex' 'concave' first on whether the function f (x) is con- vex or concave, based on the definitions and your observation the function. You can generate matplotlib code to visualize the function. If you can get the answer, please reply with ANSWER: <your answers, extract the final answer in ANSWER: <final answer> and ends with TERMINATE in the RESULT.
Answer:
Figure 7: Prompt for the Math Convexity task. We follow the similar prompt format to g except
prompting the models to write the code to generate images.
22

---

PROMPT
You are given adjacency matrix of a graph and two query nodes. YOUR TASK is to find if there is path between the two nodes.
an undirected graph G, two vertices and are called connected if contains a path from u to v. A path in a graph is finite sequence of edges which joins sequence of vertices. In the query example, the nodes and the adjacency matrix are zero-indexed.
Definition of connectivity:
Query Example:
Adjacency Matrix:
0
0
0
0
0
07
0
0
0
0
0
0
0
0
1
0
1
0
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
1
0
0
0
0
0
Query nodes indices (zero-indexed); 4 and 0 Respond with 'yes' 'no' first on whether the query nodes are connected not in the graph. If there is a path, first provide the path as a sequence of vertices (nodes), and then explain your rea- soning. You can use networkx to draw the graph. If there is no path, explain why in details. Answer (start with 'yes' or 'no'): If you can get the answer, please reply with ANSWER: <your answer>, extract the final answer in FINAL ANSWER: <final answer> and ends with TERMINATE, in the RE- SULT.
Answer:
Figure 8: Prompt for the Graph Connectivity task. We follow the similar prompt format to g except prompting the models to write the code to generate images.
23

---

PROMPT
You are given visual representation of two graphs, graph G on the left and graph H on the right. YOUR TASK is to determine whether the two graphs are isomorphic to each other.
Defimition of graph isomorphism: In graph theory, an isomorphism of graphs G and H is a bijection f between the vertex sets of G and H, denoted as f V(G) - V(H). G and H are said to be isomorphic when satisfies the following: any two vertices and e of G are adjacent in G if and only if f(u) and f(e) are adjacent in H. This kind of bijection commonly described as "edge-preserving bijection", in accordance with the general notion of isomorphism being paructure-preserviee bijection. In the query example, the adjacency matrices are zero-indexed. Adjacency Matrix G:
0
0
0
0
0
0
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
0
0
1
0
1
0
0
1
00
0
Adjacency Matrix H:
T
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
Respond with 'yes' 'no' first on whether the two graphs are isomorphic to each other. You can use networkx draw the graph. they are isomorphic, first provide the bijection between the graphs, and then explain your reasoning. You can use networkx to draw the graph. they are not isomorphic, explain why in detail. Answer (start with 'yes' or 'no'): ): If you can get the answer, please reply with ANSWER: <your answer>, extract the final answer in FINAL ANSWER: <final answer> and ends with TERMINATE in the RESULT. Answer:
Figure 9: Prompt for the Graph Isomorphism task. We follow similar prompt format to except prompting the models to write the code to generate images.
24

---

PROMPT
You given adjacency matrix of graph and two query nodes (one source node and sink node). The source node is the node where the flow starts and the sink node is the node where the flow ends.
YOUR TASK is to solve the maxflow problem given the weighted directed graph.
Definition of Maxflow probiem: In the max flow problem, we have directed graph with source node and sink node and each edge has capacity (integer valued, colored in green) that represents the maximum amount of flow that be sent through The goal is to find the maximum amount of flow that can be sent from 8 to I, while respecting the capacity constraints on the edges.
Query Example: Adjacency Matrix:
0
1
4
0
0
0
0
Source node (zero-indexed);
Sink node (zero-indexed): 2
In the query example, the nodes and the adjacency matrix are zero-indexed. You can use networkx to draw the graph. If you can get the answer, please reply with AN- SWER: <your answer>, extract the final answer in FINAL ANSWER: <final answer> and ends with TERMINATE in the RESULT.
Answer:
Figure 10: Prompt for Graph Maxflow task. We follow the similar prompt format to prompting the models to solve the maxflow problem.
8 except
PROMPT
Given the following FEN of the chess game:
(aarrk1/IbbblQppp htpp:///pnn////PPPN/// Ptt///PP//IPPPPPRRRT b--018
Determine the game's 's outcome. Who won: White or Black? Answer (start with 'white' or 'black' or
'draw'):
You can draw the chess board using Python given the FEN string. If you can get the answer, please
reply with ANSWER: <your answer>, extract the final answer in FINAL ANSWER: <final answer> and ends with TERMINATE in the RESULT. Answer:
Figure 11: Prompt for Winner ID task. We follow similar prompt format except prompting the models to analyze the game outcome.
25

---

D Dataset Statistics
Table
and
6 show the statistics of the datasets we used, including IsoBench 8
BLINK
9
MMVP
and Bench [5]]
Dataset
size
partition
representation
Math Parity
383
val
code
Math Convexity
255
val
code
Graph Maxflow
128
arry
Graph Connectivity
128
arry
Graph Isomorphism
128
val
arry
Winner ID
257
val
FEN
Table 5: IsoBench
g data statistics.
Datasct
size
partition
input
V'+Bench
257
Single Image
MMVP
300
Single Image
BLINK Relative Depth
124
val
Single Image
BLINK
Spatial Relation
143
Single Image
BLINK
Jigsaw Puzzle
150
Multiple Images
BLINK
Visual Correspondence
172
val
Multiple Images
BLINK Semantic Correspondence
139
val
Multiple Image
Table 6: Vision tasks data statisties.
Costs
The cost of running each task using GPT-4o is in Table 7
Dataset
tokens per sample
GPT-4o cost per sample
Math Parity
2994
$0.015
Math Convexity
2211
$0.011
Graph Connectivity
2819
$0.014
Graph Isomorphism
3143
$0.016
VeBenchh
26647
$0.133
MMVP
11870
$0.059
BLINK Relative Detph
14078
$0.070
BLINK Spatial Relation
12848
$0.064
BLINK Jigsaw Puzzle
13206
$0.066
BLINK Visual Correspondence
16988
$0.085
BLINK Semantic Correspondence
11508
$0.058
Table 7: The cost of running SKETCHPAD on each task.
Impact Statement
Our work proposes SKETCHPAD, a framework aiming at advancing academic research and meeting industry needs. In a broader perspective, SKETCHPAD proposes new way that humans can interact with LMs, and makes LMs more interpretable by eliciting their reasoning process with both language and sketches. On the other hand, if misused, the LMs may be used to generate harmful vision and text artifacts. Nevertheless, this is not directly related to our research, and more researchers can be involved research the safety issue in multimodal context.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
Both the abstract and introduction reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
We do not make any theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in $ and C
and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[No]
Justification:
The dataset is accessible online. We are committed to releasing the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisiioooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The test details are specified in D and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification; Since our work focus on inference of language models, the results are provided as evaluation scores on validation/test set. And we have explained how they are calculated in 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of computing resources like compute device, number of workers, memory and time of execution
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics, and strictly followed it this
work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we talk about boarder impacts in Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focus inference of language models, and thus there is no data models to release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes. We acknowledge the codebases we have referred to, cite and provide the urls of datasets and models in ??.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "5LCbtGnMo1", "reviewer": "Reviewer_BpTp", "summary": "The Visual SKETCHPAD framework integrates sketching capabilities into multimodal language models, enabling them to iteratively draw, plan, and reason using visual artifacts - similar to how humans leverage sketching to facilitate problem-solving. Unlike prior approaches that relied on text-to-image models, Visual SKETCHPAD allows language models to directly create visual elements like lines, boxes, and marks, and can even incorporate specialized vision models to enhance the sketching process. Experiments across a range of math and complex visual reasoning tasks demonstrated that Visual SKETCHPAD substantially improves performance.", "questions": "More explaination about the human evaluation is needed. I noticed the author have included the user study. How exactly the user study is conducted? The author is suggested to elablorate this.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Comprehensive Experiment: Evaluated on a wide range of math and complex visual reasoning tasks and achives decent results, which is good. \n2. Good Idea: Demonstrates the value of integrating sketching and visual reasoning capabilities into multimodal language models.\nBrings language models closer to how humans naturally think and solve problems using a combination of language and visual artifacts.\n3. Well delivery. The paper is well-written.", "weaknesses": "More discussion about the robustness (repeatibility) is needed, since a common problem for these commercial LVLMs is the instability. Does the method alleviate the invalid responces? How robust is it? Does multiple re-try helps to improve the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind and insightful feedback! We are honored that you believe Sketchpad is a great idea. We address your questions as follows:\n\n**1. More discussion about the robustness (repeatibility) is needed, since a common problem for these commercial LVLMs is the instability. Does the method alleviate the invalid responces? How robust is it? Does multiple re-try helps to improve the performance?**\n\n**Answer:** This is a great question. Regarding invalid responses, we observe that GPT-4-turbo and GPT-4o rarely produce invalid responses for our datasets. There are a few (<5) examples per dataset blocked by safety shields. For robustness, we set the temperature of decoding to 0 to reduce randomness, but still notice some instability in the OpenAI API. To address this, we conducted 3 runs per task. For instance, on BLINK depth, the GPT-4o baseline has a standard deviation of 1.5%, while with visual sketchpad, it's 1.2%. We will update the final version with mean and variance for 3 runs for each task.\n\n**2. More explaination about the human evaluation is needed. I noticed the author have included the user study. How exactly the user study is conducted? The author is suggested to elablorate this.**\n\n**Answer:** For human evaluation, we present the reasoning steps of GPT-4o (interleaved between text and images) to two human subjects. In the geometry task, we ask the annotators to respond with \"yes\" or \"no\" to the question: \"Would you draw the same auxiliary lines to answer the question?\" For computer vision tasks, we ask: \"Are the visual reasoning steps reasonable?\" We will include an additional human evaluation section in the appendix to provide more comprehensive details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "csw12oVw9R", "reviewer": "Reviewer_nzNU", "summary": "This paper studies the problem of using language models to generate code to draw for intermediate reasoning. Particularly, the idea of chain-of-thought is applied to facilitate the reasoning process, such that the auxiliary \"drawings\" enhance the LM's reasoning ability. The proposed method is tested both on math and vision tasks, showing promising results.", "questions": "- Is there any evidence that LMs can change the plan given the intermediate visual outcomes during reasoning?\n- Is there an in-context learning for LM-based code generator?\n- Is that possible to equipt CoT to VisProg (using the same specialist vision models of this work), thus achieving similar performances on math and vision problems?\n- To solve the geometry problem, the proposed model is going to generate auxiliary lines. Is there GT that can be used to quantitatively assess the correctness of the lines?\n- In Table 3, VisProg performs poorly and the reason claimed is the errors from vision modules. Why Sketchpad did not suffer from this issue?", "limitations": "Adequately discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Chain-of-thought (CoT) is introduced in the reasoning process. Moreover, visual elements, such as lines, boxes, marks, and segmentation maps, are used for intermediate reasoning steps. In contrast, most previous works only use text.\n- The overall pipeline is reasonable, and good results are achieved on different problems, including math and vision tasks.\n- The paper is well-organized and written.", "weaknesses": "- The overall idea of using LM to generate code to manipulate images is not new. As the authors pointed out, VisProg is one of the most similar works. Personally, this work could be an incremental work based on VisProg by introducing chain-of-thought and applying multimodal reasoning.\n\n- It is very confusing to use the word \"sketches\" since the drawings are straight lines, boxes, or marks, which are in fact irrelevant to sketches from my view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback! We are honored that you believe the pipeline is reasonable, achieving good results, and the paper well-written. We address your questions below. Hope that we addressed your concerns, and we would be grateful if you could consider improving the rating after seeing our responses!\n\n**1. The idea is not new. VisProg is a similar work. Personally, this work could be an incremental work based on VisProg by introducing chain-of-thought and applying multimodal reasoning.**\n\n**Answer:** Great to hear that you are also following this direction! We have also followed this direction for a long time, and believe Sketchpad fixes the key pain point of VisProg/ViperGPT. The huge performance gain demonstrates the significance of our work. This work actually starts with the authors carefully investigating ViperGPT's trajectories when solving OK-VQA. We find that many LLM-generated programs are wrong, and the vision tools break frequently. We realize that the LLM can be much more powerful if it can investigate the intermediate visual artifacts during the execution of the programs. We further develop this idea and find that it also applies to math and geometry problems. The key innovation is not about vision tool-use, but about multimodal reasoning: LLMs should think step by step across modalities, just like humans do when they are drawing on sketchpads.\n\n**2. It is very confusing to use the word \"sketches\" since the drawings are straight lines, boxes, or marks, which are in fact irrelevant to sketches from my view.**\n\n**Answer:** We understand your concerns about our terminology. We use the term ‘sketch’ as a metaphor for the process of humans drawing things on a sketchpad while thinking. We are also open to advice on new terminology!\n\n**3. Is there any evidence that LMs can change the plan given the intermediate visual outcomes during reasoning?**\n\n**Answer:** Yes, there is substantial evidence supporting this.  For example, in the dataset V*,for 26% of the examples, GPT-4o find that GroundingDINO fails to detect small objects, and decides to use a sliding window to do a more careful search. Figure 1 (d) is another good example. After using segmentation, GPT-4o decides to use depth estimation to further confirm the answer. Furthermore, the performance gap between VisProg and our method in table 3 quantitatively shows the difference. VisProg can be viewed as a version in which the plan cannot be changed. And its performance is much lower than ours (e.g., 17% v.s. 86% on MMVP).\n\n**4. Is there an in-context learning for LM-based code generator?**\n\n**Answer:** Yes. For computer vision tasks, all tasks share the same prompt, and the prompt contains 6 in-context examples. For math tasks, there are 5 in-context examples.\n\n**5. Is that possible to equipt CoT to VisProg (using the same specialist vision models of this work), thus achieving similar performances on math and vision problems?**\n\n**Answer: ** In our VisProg experiment, to make a fair comparison, we replaced the LLM to GPT-4o/turbo, and then replaced the vision experts with the ones used by our method. It is way worse than our method, as shown in Table 3. So, the short answer is no, VisProg cannot achieve similar performance. The key difference is that in VisProg, the LLM makes the plan after seeing the textual query, without any visual context. We manually inspect VisProg’s traces, and find that many plans are not reasonable and not robust. For example, GroundingDINO often makes mistakes, and the wrong bounding boxes destroy the VisProg programs.\n\n**6. To solve the geometry problem, the proposed model is going to generate auxiliary lines. Is there GT that can be used to quantitatively assess the correctness of the lines?**\n\n**Answer:** Good question! The short answer is no because it is really hard to generate GT for a geometry problem. There are often multiple solutions for the same question, using different auxiliary lines that get the same correct answer. To evaluate the correctness of the lines, we have to show the reasoning trajectory of GPT-4o to humans and let them decide. As shown in L293-297, humans find that for 80% of the cases, the auxiliary line is reasonable.\n\n**7. In Table 3, VisProg performs poorly and the reason claimed is the errors from vision modules. Why Sketchpad did not suffer from this issue?**\n\n**Answer:** Great question! The reason is that Sketchpad includes visual results as part of the reasoning process, allowing GPT-4 to inspect the results and find the errors from vision modules. For example, GroundingDINO may make mistakes for object detection. In VisProg, since the plan is not changed, there is no way to fix it. For Sketchpad, GroundingDINO will also visualize its results, with bounding boxes drawn around the objects. GPT-4 can easily figure out which boxes are wrong by inspecting the visualized results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d0zbToU9LL", "reviewer": "Reviewer_2LyD", "summary": "This work proposes Visual SKETCHPAD, a framework designed to incorporate visual reasoning into chain-of-thought and tool-use paradigms. Specifically, a multimodal LLM (Large Language Model) addresses a query by (1) generating a plan, (2) executing an action, (3) updating the current context with the result of the action, and (4) iterating until sufficient information is gathered to answer the query.\nThe plan can include various types of visual reasoning, such as visual program generation via matplotlib and networkx, leveraging specialist vision models like object detectors, segmentation models, and depth estimation models. It also includes the use of specialized tools such as image overlay or a chess drawing Python library. The main contribution of Visual SKETCHPAD lies in the reuse of intermediate visual outputs as additional observations, which enable effective chain-of-thought reasoning. SKETCHPAD demonstrates significant performance improvements compared to text-only chain-of-thought LLMs across multiple tasks.", "questions": "- Could the authors elaborate further on the extent of their contribution and how their work is different from [a] and [60]?\n\n- Authors could include more details about baselines reported on Table 1. Where these follow a similar text-based CoT pipeline?", "limitations": "The limitations were properly addressed by the authors.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- *Originality:* Incorporation of visual intermediate steps is a natural extension of chain-of-thought reasoning for multimodal LLMs.\n\n- *Quality:* SKETCHPAD is a generalized framework for extending chain-of-thourgh reasoning with visual artefacts produced by vision models, program execution, etc.  A comprehensive evaluation on existing closed-source and open-source LLMs is provided. Authors additionally conduct a human study on the discrepancy between LLM and human made plans.\n\n- *Clarity:* The manuscript is of high quality and easy to follow and the proposed idea is simple to grasp. Authors include prompt examples on supp that clarify the practical form of the proposed framework\n\n- *Significance:* Authors demonstrate a strong performance increase for LLMs prompted via SKETCHPAD compared to text-only chain-of-thought reasoning. The proposed framework does not require any training or tuning and existing LLMs can directly be used with SKETCHPAD style prompts to solve problems that require the described type of visual reasoning.", "weaknesses": "- *Clarity:* The reviewer found the title and general positioning of this work slightly misleading. Visual SKETCHPAD naturally would refer to chain of thought framework with that includes generalised sketching capabilities. Troughout this work sketching soley refers to drawing of auxiliary lines for geometric problems and is only one of the multiple tools that the proposed framework makes use of for visual artefact generation.\n\n- *Originality:* The main concern of the reviewer is with the extent of the novelty of this work. The single differentiating factor compared to standard pipelines is the incorporation of visual artefacts on intermediate CoT steps. Incoorporation of visual reasoning information for LLM reasoning cannot be considered a purely novel proposition (for example [a] or [60]). Even though the proposed approach departs from these methods by iteratively updating the context with newly formed visual artefacts, the reviewer is still skeptical w.r.t the extent of the contribution for NeurIPS standards.\n\n[a] See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning. Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, Chuang Gan", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback! We are encouraged that you acknowledge the originality, quality, clarity, and significance of our work. We address your concerns as follows, and hope that they can clarify your concerns, and hope that you can improve the rating after seeing the responses!\n\n**1. Clarity: The reviewer found the title and general positioning of this work slightly misleading. Visual SKETCHPAD naturally would refer to chain of thought framework with that includes generalised sketching capabilities. Troughout this work sketching soley refers to drawing of auxiliary lines for geometric problems and is only one of the multiple tools that the proposed framework makes use of for visual artefact generation.**\n\n**Answer:** We respectfully point out that this is a false statement. For vision problems, our work draws numbers and bounding boxes on objects. This is similar to the human sketching process. We circle things on an image and put numbers on it to help reasoning. Even for math problems, we believe that drawing out math functions and graphs is also a sketching process. We as humans often do that on our sketchpad during math examples. \n\n**2. Origniality: the main concern of the reviewer is with the extent of the novelty of this work. The single differentiating factor compared to standard pipelines is the incorporation of visual artefacts on intermediate CoT steps. Incoorporation of visual reasoning information for LLM reasoning cannot be considered a purely novel proposition (for example [a] or [60]). Even though the proposed approach departs from these methods by iteratively updating the context with newly formed visual artefacts, the reviewer is still skeptical w.r.t the extent of the contribution for NeurIPS standards.**\n\n**Answer:** We again respectfully disagree with the reviewer. Updating the visual context during reasoning is the *key novelty* unique to our work, which brings significant performance gain. Prior works, for example [a], [60], VisProg, and ViperGPT,  only do reasoning on text, and the performance is not great. For example, [a] uses GPT-3 and gets 44.6% on OK-VQA. An earlier work, [b], directly prompt GPT-3 with image captions, get 48% by directly prompting GPT-3 with image captions. In ViperGPT, the work that gets the best zero-shot result, but are still far from supervised state of the art. We have been following this line of work, and find the key thing missing is that the LLM cannot make new plans based on new visual contexts. Our method is the first work in this line of research that greatly outperforms all existing state of the art and unleashes the power of the best multimodal LLMs.\n\nSidenote: Thanks for bringing up [a]. We will cite it in our final version.\n\n[b] Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022). An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA. AAAI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zoJgftudUq", "reviewer": "Reviewer_jSat", "summary": "This paper proposes visual sketchpad that aims to aid the existing multimodal language models (MLMs). Specifically, the visual sketchpad serves as a task-specific prompting technique that calls tools to draw sketches for the input problem as additional context to help solve the problem. This prompt technique is like a chain of thought but in a visual way, and the tools include code generation as image drawer as well as vision specialist models. Benchmarks show that the proposed visual sketchpad frameworks can significantly boost the performance of base models (e.g., GPT-4o and LLAVA-Next) on multiple math and vision tasks.", "questions": "Please see Weaknesses section for the questions.", "limitations": "Yes, the author discusses some limitations in Section 7.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an interesting and effective framework (i.e., visual sketchpad) to assist the MLMs through chain of thought. Compared with text-only chain-of-thought prompts with external tools, the proposed visual sketchpad further expands the thought generation in a visual fashion, which is particularly suitable for certain scenarios like the geometry-based reasoning for math problems. \n\n2. The authors design multiple tools that enables visual sketchpad for various tasks: code generation to draw diagram/figure for math problems, and visual specialists (including depth generation / sliding window / zoom-in / crop / overlay images) for visual reasoning tasks. The rich tools provide multiple combinations to analyze the input problem and generate suitable sketches for problem solving.\n\n3. The proposed visual sketchpad does not need any training or finetuning, making it easy to integrate with existing MLMs. In the evalution, the GPT-4o / GPT-4 Turbo w/ visual sketchpad got significant improvement on geometry, graph, math, and game problems. In addition, the sketchpad also brings decent performance gain and achieves SOTA on several complex visual reasoning tasks.\n\n4. The paper is well written and easy to comprehend.", "weaknesses": "1. As the proposed visual sketchpad attempts to generate code to draw the images. I wonder if it is possible that sometimes code has issues and cannot generate images properly. If so, how to deal with such issues.\n\n2. For the geometry problem, can the visual sketchpad support geometry image (e.g., in jpg) as input? If so, how does the visual sketchpad overlay lines on the image?\n\n3. Currently, the visual sketchpad is a set of different prompts that tackle specific problems with a corresponding set of tools. Can the visual sketchpad be a single prompt to call tools and tackle all the problems listed in the evaluation? If so, that would be useful as the prompt can be integrated into the system prompt for MLMs as an enhanced version.\n\n4. The evaluation misses an important set of comparisions: There is no comparison between the proposed approach and the existing chain-of-thought frameworks.\n\n5. Although most benchmarks in the evaluation show that the additional context from visual sketchpad brings benefits, I wonder if there are some cases that the additional context introduces noises and causes errors compared with the baseline model.\n\n6. Typos\nLine 64: jigwaw -> jigsaw", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and constructive review! We are honored that you believe visual sketchpad is interesting and effective. We address each question as follows. Hope that our response clarified your concerns, and we would be grateful if you could consider improving the rating after seeing our responses!\n\n**1. As the proposed visual sketchpad attempts to generate code to draw the images. I wonder if it is possible that sometimes code has issues and cannot generate images properly. If so, how to deal with such issues.**\n\n**Answer:** We thank the reviewer for pointing this out. Sometimes, LMs generate incorrect code that results in execution errors. Our framework addresses this problem by feeding the error message back to the model and prompting it to generate a revised version of the code. This idea was developed by a long line of prior work, for example, self-debug [1] and AutoGen[51]. We will add more discussions in the final version.\n\n[1] Chen, Xinyun, Maxwell Lin, Nathanael Schärli, and Denny Zhou. \"Teaching large language models to self-debug.\" arXiv preprint arXiv:2304.05128 (2023). \n\n\n**2. For the geometry problem, can the visual sketchpad support geometry image (e.g., in jpg) as input? If so, how does the visual sketchpad overlay lines on the image?**\n\n**Answer:** This is a great question. We experimented with many multimodal LLMs (e.g., GPT-4o) and diffusion-based image editing models (e.g., SDXL-edit), but found that these models were unable to accurately add the required line to the geometry image. As a result, our current framework does not support using only the geometry image as input. Instead, we provide the matplotlib code for the geometry diagrams, enabling the LLM to add a line programmatically. Future work would be to develop or fine-tune an image-editing model specifically for overlaying precise lines on geometric images. We appreciate your question and will include this discussion in our future work section. \n\n**3. Currently, the visual sketchpad is a set of different prompts that tackle specific problems with a corresponding set of tools. Can the visual sketchpad be a single prompt to call tools and tackle all the problems listed in the evaluation? If so, that would be useful as the prompt can be integrated into the system prompt for MLMs as an enhanced version.**\n\n**Answer:** Thanks for your suggestion! Visual sketchpad uses a single unified prompt for all vision tasks and similarly, a single unified prompt for all math tasks. These prompts can be directly integrated into the system prompt of any MLMs.\n\n**4. The evaluation misses an important set of comparisions: There is no comparison between the proposed approach and the existing chain-of-thought frameworks.**\n\n**Answer:** Thanks for your feedback. We'd like to clarify that our GPT-4-turbo and GPT-4 baselines already incorporate chain-of-thought reasoning. These models produce multiple reasoning steps before outputting a final answer. It is widely believed that OpenAI trains their models to output chain-of-thought reasoning during the instruction-tuning and alignment processes. Here is a sample output from the baseline methods. As you can see, the baseline already employs a chain-of-thought approach:\n\n**Query:** You are given an adjacency matrix of a graph and two query nodes. \\n\\nYOUR TASK is to find if there is a path between the two nodes.\\n\\n    Example:\\nAdjacency Matrix: [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]....\n\n**Baseline GPT-4 output:** The adjacency matrix provided represents an undirected graph where each row and column corresponds to a node, and the value at a given position (i, j) indicates whether there is a direct edge between node i and node j. A value of 1 represents an edge, and a value of 0 represents no edge.\\n\\nFrom the adjacency matrix, we can see that most nodes have no edges connecting them to any other nodes, except for nodes 8 and 10 which are connected to each other (as indicated by the 1s at positions (8, 10) and (10, 8)).\\n\\nThe query is asking whether there is a path between node 9 and node 4. Looking at the adjacency matrix:\\n- Node 9 has no edges connecting it to any other nodes (the entire row and column corresponding to node 9 are zeros).\\n- Similarly, node 4 has no edges connecting it to any other nodes (the entire row and column corresponding to node 4 are zeros).\\n\\nSince neither node 9 nor node 4 has any connections to other nodes, and there are no intermediate nodes that connect these two nodes, there is no path between node 9 and node 4.\\n\\nFINAL ANSWER: no\\n\n\n**5. Although most benchmarks in the evaluation show that the additional context from visual sketchpad brings benefits, I wonder if there are some cases that the additional context introduces noises and causes errors compared with the baseline model.**\n\n**Answer:** This is a good point. Yes. There are instances where the additional context from the visual sketchpad can introduce errors. For example, the vision expert in our framework such as GroundingDINO, may wrongly annotate or miss bounding boxes. We found that GPT-4o is really good at figuring out if these vision experts are making mistakes, and correcting them during reasoning, as we write in L 288-289. But there are times where GPT-4o is still misled by these noises. We appreciate your feedback and will include this point in the limitations section of the final version. \n\n**6. Typos Line 64: jigwaw -> jigsaw**\n\n**Answer:** Thanks for pointing out the typo. We will fix it in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their timely and positive feedback. We are encouraged that the reviewers believe visual sketchpad is “a good idea” (Reviewer BpTp), “interesting and effective” (Reviewer jSat), with “originality” and “significance” (Reviewer 2LyD). Also, all reviewers believe that the experiments are “comprehensive” and achieved great results; the paper is “well-written” with “great clarity”. (Reviewer BpTp, nzNU, 2LyD, jSat)  We believe visual sketchpad is a novel and effective framework to fully unleash the power of multimodal LLMs, “bringing LMs closer to how humans naturally think” (BpTp) by solving complex problems with a combination of text and visual artifacts.

We will address each reviewer’s questions below. And we are happy to address any further comments from reviewers.

Meta-review: BpTp considers the problem useful/well motivatated and experiments to be comprehensive, covering diverse reasoning tasks.   They ask about determinism of the response and accuracy of responses and receive a satisfactory response in the rebuttal.  The score is maintained at accept (7)

nzNU recognizes the paper contributes to a research trajectory, and asks whether the CoT contribution could be built upon one such technique (VisProg).  A minor point is made re: the use of the terminology ‘sketch’ in this context.  Some questions are asked regarding the technical exposition and about the ground truth for geometry problems.  In my interpretation there are no serious concerns raised by the reviewer only some questions raised out of interest, and the rebuttal has answered them. The reviewer does not reply to the rebuttal so the final score is borderline accept (5).  

2LyD is concerned that introducing the modality of ‘sketch’ i.e. diagrams into the chain of thought is not a novel step over existing chain of thought-like approaches.  They also had some questions on how the work is positioned, partially echoing nzNU.  They specifically ask for clarification of novelty over [See et al] and [60], and on the baseline comparisons.  .  There is some reply given around novelty and performance relative [60] and [a] and it should be noted these papers are in arxiv/unpublished state.  The reviewer does not reply to the rebuttal and the score remains at (5).

jSat considers the framework to be interesting and effective, noting the performance gain and SOTA results.  The main weakness identified is lack of comparison with existing CoT frameworks.  The majority of the questions raised are clarification on image formation and edge case behaviour.  The reviewer notes that ‘most’ of their concerns are addressed but has not amended their score of (5).
This is a borderline paper.  No reviewer has argued against acceptance, but only one reviewer is firmly arguing to accept it. The main deficiency in the paper is the lack of comparison to other published visual CoT methods, as the reviewers flagged.  In my view the paper could do better to motivate/position itself relative these works (also see Chen et al. AAAI 2024) as there is an implication in the motivation that this work is first out of the gate on the topic.  The baseline comparisons requested would be interesting but not all of the techniques discussed by the reviewers are published yet.

On the balance the score is low but I don’t see any critical concerns raised by the reviewers that remain post-rebuttal.  I would consider that the paper is borderline acceptable on that bass and so recommend acceptance.
Decision: Accept (poster)