ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding
Authors: Xingyu Fu
Minqian Liu
Zhengyuan Yang
John Richard Corring
Yijuan Lu
Jianwei Yang
Dan Roth
Dinei Florencio
Cha Zhang
Venue: ICML 2025 poster (2025)
Keywords: Visual Chain of Thought
Visual Prompting
Table
Chart
Vision Language Model
Multimodal LLMs
Visual Programming
Primary area: applications->computer_vision
TL;DR: A new visual chain-of-thought method + used to collect training data + trained model, for table and chart image understanding.

Abstract: Structured image understanding, such as interpreting tables and charts, requires strategically refocusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, we introduce ReFocus, a simple yet effective framework that equips multimodal LLMs with the ability to generate ``visual thoughts'' by performing visual editing on the input image through code, shifting and refining their visual focuses. Specifically, ReFocus enables multimodal LLMs to generate Python codes to call tools and modify the input image, sequentially drawing boxes, highlighting sections, and masking out areas, thereby enhancing the visual reasoning process. We experiment upon a wide range of structured image understanding tasks involving tables and charts. ReFocus largely improves performance on all tasks over GPT-4o without visual editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth analysis of the effects of different visual edits, and reasons why ReFocus can improve the performance without introducing additional information. Further, we collect a 14k training set using ReFocus, and prove that such visual chain-of-thought with intermediate information offers a better supervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over CoT.

REFOCUS: Visual Editing as a Chain of Thought for Structured Image Understanding
Xingyu Fu
Minqian Liu?
Zhengyuan Yang² John Corring' Yijuan Lu Jianwei Yang?
Dan Roth Dinei Florencio Cha Zhang²
https://zeofuu.iitie otu
Abstract
1. Introduction
Structured image understanding, such interpret- ing tables and charts, requires strategically refo- cusing across various structures and texts within an image, forming a reasoning sequence to arrive at the final answer. However, current multimodal large language models (LLMs) lack this multihop selective attention capability. In this work, introduce REFOCUS, a simple yet effective frame- work that equips multimodal LLMs with the abil- ity to generate "visual thoughts" by performing visual editing on the input image through code, shifting and refining their visual focuses. Specif- ically, REFOCUS enables multimodal LLMs generate Python codes to call tools and modify the input image, sequentially drawing boxes, high- lighting sections, and masking out areas, thereby enhancing the visual reasoning process. We ex- periment upon a wide range of structured image understanding tasks involving tables and charts. REFOCUS largely improves performance on all tasks over GPT-4o without visual editing, yield- ing average gain of l 11.0% on table tasks and 6.8% on chart tasks. We present an in-depth anal- ysis of the effects of different visual edits, and reasons why REFOCUS can improve the perfor- mance without introducing additional information. Further, we collect a 14k training set using REFO- CUS, and prove that such visual chain-of-thouuhh with intermediate information offers better su- pervision than standard VQA data, reaching a 8.0% average gain over the same model trained with QA pairs and 2.6% over COT.
Structured images, such as tables and charts (Kafle et al., 2018; Chen et al., 2020; Zhu et al., 2021; Masry et al., 2022; Kim al., 2024: Wang et al., 2024), are essential for the efficient communication of information in our daily lives (Chang et al., 2008; Groom et al., 2016). Understand- ing these structured images requires multiple reasoning steps each paying selective attention (Duncan, 1984; Johnston & Dark, 1986; Rizzolatti et al., 1994) to focus on particular pieces of related information while ignoring less pertinent details and distractions, facilitate multi-hop visual rea- soning and arrive the final answer. For instance, What's total wins by Belgian riders in Figure 1? To answer this, we might 1) identify the teams with Country being Belgium, 2) find the wins of each team, 3) locate the wins of the teams from Belgium, and 4) sum the wins together.
One major limitation of current multimodal models is their lack of selective attention and multi-hop visual reasoning ability they limit intermediate reasoning to textual for- mats only. Most methods often extract the information from the image to text first, and then rely on language model's chain-of -fhuuhh (CoT) (Wei et al., 2022) reasoning to solve the real problems (Masry et al., 2023; Han et al., 2023; Liu et al., 2023e; Zhang et al., 2023; Chen et al., 2024; Surís et al., 2023; Gupta Kembhavi, 2023), never looking back at the image again. Recent work Visual Sketchpad (Hu et al., 2024) for the first time attempts to create visual artifacts as thoughts using vision tools. However, it only works on natu- ral images and mainly benefits from additional information brought by tools instead of from visual reasoning.
In this work, we explore an improved representation of in- termediate thoughts, to boost the visual reasoning ability of multimodal large language models (LLMs) on structured image understanding. We demonstrate that simple visual editing actions generated as Python code from multimodal LLMs-such as drawing boxes, highlighting areas, or mask- ing regions-can can significantly improve model performance by directing selective attention.
WWrk done during internship at Microsoft. Universsty of Penn- sylvania *Virginia Tech "Microsoft. Correspondence to: Xingyu Fu <xingyuf2 @ seas. weran..uenneeuu.. edu>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To this end, we introduce REFOCUS: framework that en-

---

REFOCUS
What's total wins by Belgian riders ?
THOUGHT: Focus columns Country and Wins
09 THOUGHT: Focus with Country-
ACTION: Focus with Mask using Python Code
ACTION: Focus with Draw using Python Code
ANSWER:
Focus calamne kith draudig focce,
Eocaue comuur C'curntyy "Nare's
sucao Mooc all Moor cow0, ]
"3"3
mask tot_aos cor_col
Nusi zur Eecetavent osteamss zecCangtetmas
#lccaa a wous gullarans 2003 www.://tteggatteeueeeoooo
editing on an example data from TQA (Kim et al., 2024). Given an image and question pair, REFOCUS equips GPT-4 with editing tools (details in $3), and GPT-4 generates pscudo code if an edit action needed. REFOCUS then executes the editing actions, and feeds GPT-4 with the new image until an answer is reached. In the above example, mask_column and draw_row are performed.
Figure 1. Overview of REFOCUS. REFOCUS performs visual chain of thought via
hances multimodal L.Ms by integrating visual reasoning as an intermediate step. REFOCUS provides an interface that allows models to generate visual artifacts with set of image editing tools implemented in Python code. Specifi- cally, REFOCUS prompts the underlying multimodal LLM to program, executes the code, and produces visual artifacts that incorporate selective attention as the new input for the model. For instance, for long and complicated tabular im- age as in Figure 1, REFOCUS masks irrelevant columns and highlights important rows by drawing boxes around them. Similarly, find "the average of last four count ries data" in Figure 2, REFOCUS modifies the figure by remov- ing the data of other countries. The new image therefore climinate distractions, avoiding possible hallucinations, and become easier to solve for multimodal LLMs by refocusing.
Further, we investigate whether we can distill such refocus abilities to smaller models through supervised-finttuniig (SFT), and if such visual chain-of- thoutht (VCoT) data could benefit models more than the standard visual ques- tion answering (VQA) data. We collect 14k training set data including the focus area bounding boxes and reasoning processes using REFOCUS + GPT4o. Surprisingly, com- prehensive experiments on Phi-3.5-vision show that our collected REFOCUS data serves a better supervision signal than standard VQA data for SFT, achieving an average gain of 8.0 % over the same model trained with the same QA data, and 2.6% % over CoT data.
To summarize, we introduce (1) REFOCUS, a simple yet effective visual reasoning framework that enhances struc- tured image understanding through input-image editing; (2) we demonstrate that REFOcUS consistently achieves per- formance improvements and analyze underlying reasons behind these gains; (3) we curate a 14k training set using REFOCUS and GPT-4o, and show that model finetuned with REFOCUS data consistently outperforms the same model trained with the same set of QA or CoT data, suggesting potential pathways for more intelligent reasoning in vision language models.
We demonstrate the effectiveness of REFOCUS across a wide range of structured image tasks, focusing on table and chart visual question answering (VQA) sets. For tabular problems. REFOCUS enables models to selectively edit the columns and rows in the input image. REFOCUS consis- tently improves the baseline GPT-4o performance, yielding an average gain of 11.0%. % For chart problems, we tackle diverse types of charts including (1) horizontal bar charts, (2) vertical bar charts, and (3) complex scientific charts from arXiv papers. REFOCUS empowers multimodal LLMs to modify bars and subplots to pay selective attention, resulting in consistent improvements across different types of chart images, with the average gain over got-4o reaching 6.8 % We present an in-depth analysis on REFOCUS. We first study why REFOCUS could achieve large gains, especially since does not bring in external information as other meth- ods (Hu al., 2024) do, and examine how different editing techniques could affect multimodal LLMS differently.
2. Related Works
Structured Image Understanding Scientific chart and table figures have long been challenging task for mod- ern multimodal models. Previous methods mainly utilize Optical Character Recognition (OCR) (Liu et al., 2024b), by turning the image into text format and then performing textual reasoning (Liu al., 2022; 2023a). Other methods (Masry al., 2023; Han al., 2023) focus enhancing
2

---

REFOCUS
What is the average of last four countries data ?
European Millennials Say Success Out of Their Hands
European Millennials Say Success Is Out of Their Hands
% of Millennials who say success life is pretty much determined by forces outside our control
% of Millennials who say success in life is pretty much determined by forces outside our control
U.S.
Germary
Ioaly
Greece
Poland
Poland
Frence
France
Spain
Spain
UK
Soure: Spring 2014 Global Artinudee munveyy.
Source: Spring 2014 Global Auttudes survey.
oo
GPT-4o: Last four countries: Greece 62%, Poland 62%, France 54%, and Spain 48%. So average is 56.5.
00
GPT.4o: The last four countries: Poland: 62, France: 54, Spain: 48, UK: 37. So the average is 50.25.
Figure Example of how REFOCUS + GPT-40 solves previously unsolvable problem ChartQA dataset (Masry et al., 2022) through improved visual grounding. Given the original horizontal bar image (left), GPT-40 grounds to the wrong bars and thus gets the wrong answer. REFOCUS eliminates such possibility through editing, guiding the model to the correct answer (right).
end-to-end VQA capabilities by training on augmented data.
demonstrating that segmenting the original image can signif- icantly enhance the visual grounding capabilities of GPT-4v models. Meanwhile, it has been highlighted in BLINK (Fu et al., 2024b) that most open-source multimodal language models may struggle to comprehend visual prompts. Recent works, such as VIP-LLaVA and List Items One by One (Cai et al., 2024; Yan et al., 2024), have further refined these models to improve their understanding and processing of visual prompts. Most of these visual prompts are visual objects centered and do not apply to structured images. One recent work Visual Cot (Shao et al., 2024a) extracts the subarea in images where the answer corresponds to, e-g. ob- taining the coordinates of answer text using OCR chart VQA problems, or find the answer object's coordinates in the images using object detection methods. However, Visual Cot do not involve multistep reasoning and refocusing pro- cesses and only focuses on image region-answer text direct mapping. We compare our results with Visual Cot on Chart QA datasets in Section 5 and show absolute improvement.
Visual Reasoning through Programming A significant recent trend in multimodal LLMs is using Python programs to facilitate chain-of-thougle reasoning (Gupta & Kembhavi, 2023; Surís et al., 2023). However, these methods typically conduct reasoning at the text level (Fu et al., 2023; 2022: 2024a). While some methods may utilize image crops, the images themselves remain unchanged, which limits their visual reasoning capabilities. More recent work, such Vi- sual Sketchpad (Hu et al., 2024), advances visual reasoning by providing various computer vision tools like Segment Anything, DINO, and Depth Anything (Yang et al., 2024; Li et al., 2023; Liu et al., 2023d) to create visual artifacts. However, Visual Sketchpad mainly achieves performance gains by incorporating external expert knowledge through these tools, whereas explore the possibility of enhancing visual understanding without additional information. Also, Visual Sketchpad cannot solve text-rich images in struc- tured image problems, as its tools are vision-based and only address object-centric issues. With REFOCUS, LLMs can write code to call different functions for editing the input im- age, sequentially simplifying the original problem to make it easier for multimodal L.MSs to solve. This approach enables better visual understanding and reasoning without relying on external data or expert knowledge.
3. REFOCUS
REFOCUS conducts visual editing on input images as a chain-of-thoughi providing an intermediate interface for multimodal LLMS to facilitate visual reasoning and enforce the correct answer with a valid reasoning process. The entire pipeline is iterative, illustrated in Figure 1: the mul- timodal LLM sees the image and question, proceeds with one step of thought, conducts visual editing using Python code, and continues with the next step of thought and editing until it arrives at the final answer. We specifically target two sets of structured image problems in this work: (1) tabular figures, and (2) chart figures. This section unfolds by intro- ducing the details of our tasks ($3.1), 1), the implementation of
Visual Reasoning through Visualization It has become evident that certain types of visual prompts the input images, such as adding red circle around a target object, can significantly impact the performance of multimodal lan- guage models (Shtedritski et al., 2023), on certain abilities such as visual grounding (Shao et al., 2024a). The Set-of- Mark study (Yang et al., 2023) extends this investigation by
3

---

REFOCUS
At what period does the Cash transfer curve have the largest "share contracting COVID-19" value cifference with Control curve
25
24:11
Food
Sa
7he 2252
Meeltor
Sor
Coarnd
Clrb mmadie
Combitart
GPT-4o: From the top-left subplot, the blue dashed line (Cash transfer) is significantly higher than the black solid line (Control) in 'Jul 21' period. So Jul 21.
GPT-4o: In Mar-May 20, difference is 0.01. In Nov 20- Jan 21, the difference is 0.02. In Jul 21, the difference 0.01. So the answer Nov 20-Jan 21 period.
Figure REFOCUS equips GPT-4 with seleetive attention. Above is an example of how REFocUs + GPT-40 solves previously unsolvable problem in ChartXiv dataset (Wang et al., 2024). Specifically, REFOCUS edits upon the original image by masking out all irrelevant information - the other three subplots that could be distracting. As result, GPT-4o is able to conduct better reasoning with the edited image, and reach the correct answer.
visual editing tools for tables ($3.2) and for charts, and how to equip multimodal LLMs with these editing tools ($3.3).
using pseudo-HTMe HTML converted from the textual table data. There are total of 250 Visual Question Answering (VQA) pairs in this dataset.
3.1. Structured Image Problems
Coordinate Acquisition for Tables. To support visual edit- ing, we need to acquire coordinates for each column and row. We accomplish this using the openev-python package with functions such as findContous()) and getStructuringEEle men((). The heuristic behind this process is that we detect lines and boxes around text in the figure. The longest verti- cal line should correspond to the row length, and the longest horizontal line should correspond to the column length. By combining the longest contours in the figure, the coordi- nates of text boxes, and the row and column lengths, we can determine the coordinates of each column and row.
Tabular Problems. Tabular understanding has long been a challenging task for multimodal LLMs (Liu et al., 2022: 2023a). In this paper, we use TQA (Kim et al., 2024) our bed.-bed. We experiment three types of table data:
VWTQ. VWTQ is derived from WikiTableQuestion (WTQ) (Pasupat & Liang, 2015), which provides original HTML data on Wikipedia tables along with corresponding Question Answering (QA) pairs and maintain its accuracy-
based evaluation metric. (Kim et al., 2024) reproduces the original table images from WTQ by applying the stylesheet of Wikipedia to the HTML and capturing screenshots of the table images. There are a total of 750 Visual Question Answering (VQA) pairs.
Chart Problems. Chart understanding is another fundamen- tal task (Liu et al., 2022; 2023a). serving pivotal role in real-world applications such as analyzing scientific papers financial reports. We use the datasets CharXiv (Wang et al., 2024) and ChartQA (Masry et al., 2022) for our ex- periments:
VWTQ_syn Considering that the table figures in VWTQ come from Wikipedia and can easily be web-crawled to gather pre-training data for multimodal LLMs, (Kim et al., 2024) generates synthetic table images using table ren- dering system based on the WTQ data. This system takes HTML as input and generates tables with various styles, fea- turing random attributes such as background colors, border margins, and font families. There are a total of 250 Visual Question Answering (VQA) pairs in this synthetic dataset.
CharXiv Multi-subplot. CharXiv is a high-quality, expert- annotated, challenging VQA benchmark that involves natu- ral and diverse charts from arXiv papers. CharXiv includes two types of questions: descriptive questions that examine basic chart elements and reasoning questions that require synthesizing information across complex visual elements in the chart. In this paper, we specifically focus on the chal- lenging reasoning questions that involve multiple subplots Due to computation cost concerns, we randomly select 200
VTabFact. TabFact (Chen et al., 2019) represents a verifi- cation task that determines whether a statement is entailed or refuted given table data in text format. (Kim et al., 2024) proposes a rendering system that generates visual tables

---

REFOCUS
How many years have value less than 10% ?
GPT-4o: Looking the chart, 2011: 8.8%, 2014: 5.3%, 2018: 2.4%, 2019: -6.8%. there are years.
GPT-4o: The years with values less than 10% 2011, 2014, 2017, 2018, and 2019. So years in total.
Figure REFOCUS unleashes better visual grounding and counting abilities for GPT-4 as in CharlQA (Masry et al., 2022) Vertical Bar problems.
VQA pairs out of the 1000 test data and use GPT-4 to select the ones with multiple subplots for experimental purposes, yielding total of 143 VQA pairs.
Highlight Column, which overlays light red color on the columns that need be focused on.
Highlight Row, which overlays light red color on the rows that need to be focused on.
Horizontal Bar. ChartQA (Masry et al., 2022) one of the most popular chart-based VQA benchmarks, consisting of human-written questions focusing on logical and visual reasoning based on ceb-crawled diverse chart figures. For our experiments, we take the horizontal bar figures and their corresponding VQA data from the test split. This subset includes a total of 444 VQA pairs.
Mask Column, which places a white mask over the columns that do not need attention.
Mask Row, which places a white mask over the rows that do not need attention.
Draw Column, which overlays solid red bounding box the columns that need to be focused on.
Vertical Bar. Similar to the horizontal bar figures, we also take the vertical bar figures along with their corresponding VQA data from the test split the ChartQA (Masry et al., 2022) dataset. This subset includes total of 382 VQA pairs.
Draw Row, which overlays a solid red bounding box on the rows that need to be focused on.
A pseudo code for example tool: Highlight Column illus- trated in Listing Visual output examples can be found in Figures and 7.
Coordinate Acquisition for Charts. For CharXiv multi- subplot figures, we acquire the coordinates of each subplot using the openev-python' package with functions such findContours() and ge(StracturingEleeenn()) ). The heuristic behind this process is that we take the top longest contours, provide them in prompt to LLMs, and allow the L.MSs to determine which coordinates correspond to which subplot (k = 10 in our experiments). Regarding ChartQA horizontal bar figures and vertical bar figures, we utilize the provided coordinates of x-axis values and y-axis values in the original dataset. Additionally,
from import
IraageDraw
def focus, on columns, with highlight columns, to. so_focus oom. all_cclumnn_ ail_colame__boodding boses]: This function
focus specilic columns the image.
e Drow highlight color on the columas to focus on mask image convert RGOpp)
Imagelheww Drawinask)
e lierate over the columns to highlight for column, name columos_to tocouttooouous_oooooooo_o
bounding
14
column, bbox all cll_uoou boundong baaaedigghhoeetcouuumm.o (xl. yl. x2, y2) column, bbox mask coctungellll
15
combine this information with co- ordinate details about the chart area (excluding the caption) obtained using the opencv-python' package.
yl). y2)), fill 50))
Composits the overlay with the mask over the original image edited jmage taagge.llpaacoonge cow.ppiiccccmmsittttitaaaaas comvert( RRGHA)) mask)
edind_jimage
3.2. Visual Editing Tools
Listing I. Python code example for highlighting column.
We adopt three types of visual editing method in REFO- CUS: mask out, draw box, and highlight color using Python code. In our experiments for tabular problems, we utilize a variety of tools as follows:
Similarly, for chart problems, we apply the same edit- ing methods, but focus on different aspects: (1) subplots, particularly for CharXiv figures; (2) bars by x values, specif- ically for ChartQA vertical bar figures; (3) bars by y values,
5

---

REFOCUS
Table
Chart
Model
vWTQ
VWTQ syn
VTabFact
CharXiv
Horizontal Bar
Vertical Bar
Prior Multimodal LMs
LLaVA-NeXT-34B (Liu et al., 2024a)
36.4
38.0
71.2
18.9
23.4
12.6
Phi vision (Abdin et al., 2024)
44.7
53.2
74.4
16.2
60.8
66.5
Gemini-Pro 1.5 (Team et al., 2023)
38.5
43.2
75.6
38.3
57.2
66.0
VisProg (Gupta & Kembhavi, 2023)
53.2
62.0
76.4
46.8
69.8
68.6
Latest multimodal L.LMs REFOCUS
gp1-4o-2024-05-13 (OpenAI, 2023)
66.5
73.2
89.6
49.0
78.2
76.2
+ REFOCUS
76.9
79.6
89.6
57.3
85.4
81.0
+10.4
+3.4
+0.0
+8.3
+7.2
+4.8
11:40--022-08-000 (OpenAI, 2023)
66.4
70.4
90.0
48.9
75.2
74.9
+ REFOCUS
77.2
82.8
90.8
46.2
82.0
81.2
49.8
+12.4
40.8
-2.7
45.0
44.2
Table 1. REFOCUS yields consistent performance gains across all tasks and outperforms all baselines. Notice that the GPT baselines here also in conversational format but without editing abilities. For fair comparison, we modify the original Visprog (Gupta Kembhavi, 2023) framework by replacing the LM and VQA components with the latest GPT-4o model.
specifically for ChartQA horizontal bar figures. Each of these aspects is combined with the three editing methods: masking, drawing, and highlighting. Detailed visual output examples can be found in Figures to 4, and complete list of tools is available Appendix
to compare with: LLaVA-NeXT (Liu al., 2024a), Phi 3 Vision' (Abdin et al., 2024), and Gemini Pro 1.5 (Team et al., 2023). In addition, we also compare our approach with the visual programming method VisProg (Gupta & Kembhavi, 2023), which uses Python code to generate in- termediate reasoning with integrated vision modules. We modify the original VisProg framework by replacing the LLM and VQA components with the latest gpt-4o-2024-08. 06 model for fair comparison. We do not include Visual Sketchpad (Hu et al., 2024) since none of its tools can apply to structured image problems, leaving it the same as vanilla GPT-4o. More implementation details are Appendix
3.3. Equip LLMs with Visual Editing Tools
We provide the function names each visual editing tool in the multimodal LLM'' prompts and ask it to generate pseu- docode using these provided function names. The actual editing functions will then be executed when the multimodal LLM decides to perform visual editing, and the modified images are returned to the model as the new input. More prompting details can be found in Appendix C.
4.2. Results
As shown in Table 1, the mean accuracy of REFOCUS com- bined with GPT-4o consistently surpasses baseline mod- els, including the vanilla conversational GPT-4o model with- out editing abilities. REFOCUS is particularly effective on VWTQ, VWTQ Horizontal Bar, and Vertical Bar tasks, achieving a consistent 5- 5-10% accuracy improvement over GPT-4o. To examine the quality of visual editings, as illus- trated Table 2, we further test LLaVA-NeXT (7B, 13B, 34) and Phi-3-vision models using the REFOCUS + GPT-40 edited images. Interestingly, while none of the tested mod- els are fine-tuned on images with visual prompts, we still observe consistent improvement VWTQ, VWTQ NWTQ_syn, CharXiv, and Vertical Bar tasks. In Table 5, we compare the performance of REFOCUS versus vanilla GPT-4o with different inputs (i.e., gold table / chart text, figure, or gold text figure). Surprisingly, GPT-4o with REFOcUS often outperforms GPTT-0 with gold text and figure inputs by a
4. Experiments and Analyses
We demonstrate that REFOCUS significantly improves mul- timodal LLMs' performance structured images, while making the visual reasoning processes interpretable. In this section, we introduce the baseline models and experiment settings ($4.1), 1), present comprehensive experiment results ($4.2), and show an in-depth analyses investing why REFO- CUS is so effective despite its simplicity($4.3))
4.1. Baselines and Setups
We apply REFOCUS on the state-of-the-ar multimodal LLM: GPT-4o (OpenAI, 2023), especially the gpt-40-2024- 05-13 and 401-40-2024408-- checkpoints show its effi- cacy. We compare with the default GPT-4o for both check- points using chain-of-thoughe prompting for fair comparison. We also include several other powerful multimodal L.LMs
"We used the checkpoint at https://huggingface co/ microsoft/Phi www-hhh-iiiionn 128k- 1-w.-iistrrccc instruct.
6

---

REFOCUS
Table
Chart
Model
vWTQ
VWTQ_syn
VTabFact
CharXiv
Horizontal Bar
Vertical Bar
Multimodal LLMs with original visual inputs
LLaVA-NeXT-78 (Liu al., 2024a)
21.7
24.0
56.8
16.1
7.3
LLaVA-NeXT-13B (Liu et al., 2024a)
25.6
30.4
62.4
18.9
13.5
13.1
LLaVA-NeXT-34B (Liu et al., 2024a)
36.4
38.0
71.2
18.9
23.4
12.6
Phi vision (Abdin al., 2024)
44.7
53.2
74.4
16.2
60.8
66.5
Multimodal LLMs with REFOCUS edited visual inputs
LLaVA-NeXT-7B Oracle REFOCUS
25.7
26.8
55.2
15.4
6.8
8.4
HaVA-NecTT-338 + Oracle REFocus
30.3
31.6
61,2
17.5
15.5
13.1
LLaVA-NeXT-34B + Oracle REFocUS
39.1
41.2
68.0
21.0
26.1
15.2
Phi vision Oracle REFOCUS
48.3
56.4
72.8
17.6
60.4
66.5
Table Open-source models' performance upon the original visual imput versus GPT-4o + REFocus edited images as visual input (referred by oracle REFocus). Notice that oracle REFOCUS uses the visual artifact generated in the last action of GPT-4o REFOcUS as inputs.
significant margin on most tasks. This demonstrates that REFOCUS effectively enhance GPT-4o's structured im- age understanding capability, making it perform as if it had access to the gold text input.
Dataset VWTQ
Original
Mask out
Draw Box
Highlight
66.4
77.2
77.6
74.8
VWTQ syn
70.4
82.4
78.8
80.8
Table 3. Analysis on how different editing tool can affect model performance. We control the tool type provided by REFoCUs and experiment on the VWTQ and VWTQ syn datasets.
4.3. Analyses
Why can REFOCUS improve the performance? This question interesting because REFOCUS does not intro- duce any additional information other methods (Yang al., 2023; Hu et al., 2024) do. Still, it achieves impres- sive performance gain upon the already powerful GPT-4o models, using simple strategy. Our observations on the in- termediate outputs suggest that this might be because REFO- CUS improves GPT-4o's visual grounding and OCR abilities through selective attention and eliminates hallucinations. As shown in Figure for the original image the left, GPT-4o mistakenly grounds Greece and misses the UK, resulting in the wrong answer of 56.5. REFOCUS decomposes the steps of visual grounding and optical character recognition (OCR). In the edited image on the right, only the last four countries are shown, and GPT-4o successfully provides the correct answer with proper reasoning. Similarily, as shown in Figure when GPT-4o tries recognize characters in the original image, it makes spelling mistake, identify- ing "Partenavia" incorrectly. However, with the highlighted columns, GPT-4o can correctly recognize the characters as "Partenvia."
edited image
all image
640
403
250
250
137
139 143
VWTQ
WWTQ_.sm
VTebFact
Charkiv
Horizontal Bar Vertical Bar
Figure 5. Statistics of how often visual editing are performed.
How frequent is visual editing performed? Since the visual editing steps are conducted through LLM generated python code, it is possible that REFOCUS does conduct any editing for some instances. To answer how often the underlying multimodal LLM, GPT-4o in case, decides to proceed with visual edit, we calculated the number of images that GPT-4o decides to edit and report the numbers in Figure GPT-4o clearly shows a preference to edit 85% VWTQ and CharXiv images, while the portion of edited images on the other datasets remains around 40-55%.
Which editing method works the best? Since we pro- vide three types of editing methods: draw box, highlight color, and mask out, raises a natural question about which method works best for the models. We conducted experi- ments on VWTQ and VWTQ sets and show the perfor- mance differences in Table 3. In short, all tools are similar.
5. Finetune with REFOCUS data
While REFOCUS shows its efficacy, it is still limited to a visual prompting method. Could the visual reasoning processes of REFOCUS be distilled into smaller Multimodal I.LMs and provide additional even better supervision for

---

REFOCUS
Edited
Image
Image
Correct Answer
ReFocus Training Set
Question
Code to Focus on Areas
Prediction
Incorrect Answer
Supervised Finetuning
Figure 6. Training set collection using REFOCUS ChartQA dataset.
models to learn? What would be the difference between model fine-tuned on the de facto image question answer pairs, versus one fine-tuned on CoT data, or on REFOcUs visual reasoning data?
Horizontal Bar
Vertical Bar
Avg.
QA Prompting
Phi-3.5-vision
60.1
63.1
61.5
SFT w/ QA Data
60.1
65.5
62.6
Visual CoT Prompting
Finetune Data Collection To answer this question, we further collect a 14k training set using REFOCUS and GPT- 4o upon the ChartQA (Masry et al., 2022) training data. As shown in Figure 6, we apply the pipeline on 15,059 ChartQA training data, If the prediction is correct, we keep the whole process and add the generated textual chain-of- thought (CoT)) reasoning, python code for editing, and the area to focus on (in bounding box format) our data. If the prediction is incorrect, we give it one more try and hint it with the gold answer. If it still gets to wrong answer, we discard this example since it's most likely a data noise from our observation. We end up with 14,344 training data, among which 12,819 has editing processes. More dataset details in Appendix D.
Visual CoT 7b-336
27.1
14.5
20.8
Phi-3.5-vision
69.4
66.8
68.2
SFT w/ QA Data
60.6
66.8
63.4
SFT w/ REFocUS CoT
67.1
70.7
68.8
SFT w/ REFOCUS VCoT
71.0
72.2
71.4
Table 4. SFT accuracy results. The difference between REFO- CUS VCoT data and CoT data is that VCoT contains refocus area bounding box coordinates whereas CoT does not. All trainings seleet the best performing hyper-parameters on the same set of training data.
Table with multiple setups the REFOCUS data is Visual CoT (VCoT) data, and compare with the standard QA data. Another setup is finetuning with CoT data, which is VCoT data without refocus area bounding box information. The Phi-3.5- Vision model finetuned with REFOCUS VCoT data outperforms the base model by 3.2% in accuracy; over the same model finetuned with same set of QA data by 8.0%; and over the model finetuned with CoT data by 2.6%. Notably, our results largely exceed Visual Cot (Shao et al., 2024b) which trains upon Va-1.115.5 1.5 (Liu et al., 2023b) with answer coordinate data by 50.6%. These results suggest that implicit visual reasoning data such as REFOCUS VCoT can consistently provide better supervision signals than standard QA pairs and CoT data for structured image understanding tasks. Our approach has the potential provide rich and meaningful supervision data enhance multimodal L.LMs in general. More SFT details, result analyses, and image editing outputs can be found in Appendices D and E.
Finetune Setups To test the effectiveness of visual chain- of-thought for multimodal LLMs, adopt the standard Supervised-Finetunnng (SFT) strategies. We use Phi-3.5- vision' (Abdin et al., 2024) as our base model, and compare its performance when finetuned with REFOCUS data, ver- sus with same set of data but only with QA pairs. For both finetuning, we keep all settings the same and search through the same set of hyper-parameters for learning rate and epoch number, and report the best performance. During training time, the input is given in the format of '<image> <question> <thought1> <ReFocus bounding box> <thought2> <answer>', where <Thought1>> is trans- formed through GPT-4o returned thought on what areas to refocus on, and <Thought2> is GPT-4o output when predict- ing based on the edited images. In comparison, the standard QA training input is '<image> <question> <answer>' During inference time, examine two types of prompt set- tings: '<image> <question> Answer: as the QA prompt- ing, and '<image> <question> Thought:' as the Visual CoT prompting. More details can be found in Appendix D.
6. Conclusion
This work introduces REFOCUS, a simple yet effective framework that boosts multimodal large language models (LLMs) understand structured images better, by incor- porating visual editing on the input image through Python code. Our approach significantly boosts performance on table and chart tasks. Further, we present a 14k visual chain-of- thouht training data built using REFOCUS, which demonstrates superior supervision over standard VQA data.
Supervised-Finetuuing Results We report our results as in
httpeithhwb.comm/ccroooo///////t 3CookBook/booob//////oooooo main/nd/84. hutnee-tuuiinniiinnnnnnnnnnnnn Vision.
8

---

REFOCUS
Acknowledgements
Fu, X., Zhou, B., Chandratreya, I., Vondrick, C., and Roth,
D. There's time and place for reasoning beyond the image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
We would like to thank Weijian Xu, Guoxin Wang, Yushi Hu and Weijia Shi for their helpful and insightful discussions. Xingyu Fu and Dan Roth were funded by ONR Contract --0014-33-1-264, ONR Contract 1-0014-23-1-24111 and NSF grant IIS-2212433.
Fu, X., Zhang, S., Kwon, G., Perera, P., Zhu, H., Zhang, Y., Li, A. H., Wang, W. Y., Wang. Z., Castelli, V., et al. Generate then select: Open-ended visual question an- swering guided by world knowledge. arXiv preprint arXiv:2305. 18842, 2023.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Fu, X., He, M., Lu, Y., Wang, W. Y., and Roth, D. Commonsense-12i challenge: Can text-to-image genera- tion models understand commonsense? arXiv preprint MXXiv:4406007544 2024a. 3
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024. 6, 8
Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., and Krishna, R. Blink: Mul- timodal large language models can see but not perceive. arXiv preprint arXiv:2404.1239 2024b. 3
Groom, C. R., Bruno, I. J., Lightfoot, M. P., and Ward, S. C. The cambridge structural database. Structural Science,
Cai, M., Liu, H., Mustikovela, S. K., Meyer, G. P., Chai, Y., Park, D., and Lee, J. Vip-Ilava: Making large mul- timodal models understand arbitrary visual prompts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12914-12923, 2024.
1712))1 20161 1
Gupta, T. and Kembhavi, A. Visual programming: Compo- sitional visual reasoning without training. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 14953-14962, 2023. 1, 3,6
Chang, F., Dean, J., Ghemawat, S., Hsich, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. E. Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems (TOCS), 26(2):1-26, 2008. 1
Han, Y., Zhang, C., Chen, X., Yang. X., Wang, Z., Yu, G., Fu, B., and Zhang, H. Chartllama: A multimodal Ilm for chart understanding and generation. arXiv preprint orXiv:2311. 16188.,, 2023. 1,2
Hu, Y., Shi, W., Fu, X., Roth, D., Ostendorf, M., Zettle- moyer, L., Smith, N. A., and Krishna, R. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. arXiv preprint
Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., and Wang, W. Y. Tabfact: A large-scale dataset for based fact verification. arXiv preprint arXiv:1909.000 164, 2019. 4
arXiv:2406.09-400 2024. 1, 3, 6,
Chen, W., Zha, H., Chen, Z., Xiong, W., Wang. H., and Wang, W. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. arXiv preprint arXiv:2004.07347, 2020. 1
Johnston, W. A. and Dark, V. Selective attention. Annual review of psychology, 1986. 1
Kafle, K., Price, B., Cohen, S., and Kanan, C. Dvqa: Under- standing data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5655 2018.
Chen, Z., Zhou, Q., Shen, Y., Hong, Y., Sun, Z., Gutfreund, D., and Gan, C. Visual chain-of-thought prompting for knowledge-based visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024. 1
Kim, Y., Yim, M., and Song, K. Tablevqa-bench: visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404.19206 2024. 1,
pp.
4, 12
Duncan, J. Selective attention and the organization of vi- sual information. Journal of experimental psychology: General, 113(5)5011, 1984.
Li, F., Zhang, H., Sun, P., Zou, X., Liu, S., Yang, J., Li, C., Zhang, L., and Gao, J. Semantic-sam: Segment and
9

---

REFOCUS
recognize anything any granularity. arXiv preprint wrXiv:2307.04767, 3
Shao, H., Qian, S., Xiao, H., Song, G., Zong, Z., Wang, L., Liu, Y., and Li, H. Visual cot: Advancing multi-modal language models with comprehensive dataset and bench- mark for chain-of- of-thought reasoning. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. 3
Liu, F,, Piccinno, F,, Krichene, S., Pang. C., Lee, K., Joshi, M., Altun, Y. Collier, N., and Eisenschlos, J. M. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering, arXiv preprint arXiv:2212.09665 2022. 2.4
Shao, H., Qian, S., Xiao, H., Song, G., Zong. Z., Wang, L., Liu, Y., and Li, H. Visual cot: Unleashing chain- of offthhough reasoning in multi molti- language models, 2024b. 8
Liu, F. Eisenschlos, J. M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., and Altun, Y. Deplot: Ohe-shot visual language reasoning by plot-to- table translation. In Findings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023a. URL httpps://aree .rgg httg//b/22/0 221 10505. 2, 4
Shtedritski, A., Rupprecht, C., and Vedaldi, A. What does clip know about red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11987-11997, 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023b.
3
Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual in- ference via python execution for reasoning. Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023. 1,
Liu, H., Li, C.,, L., B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://1lava-v/ hollog// 0022-01-38000000 100---01----1aaaaaa -0-laaaa-eexx/. 6,
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J.. Soricut, R., Schalkwyk. J., Dai, M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1800, 2023. 6
Liu, S., Cheng. H.. Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.0542 2023c.
Wang, Z., Xia, M., He, L.., Chen, H., Liu, Y., Zhu, R., Liang,
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C.,, Yang. J., Su, H., Zhu, J., et al. Grounding dino: Marry- ing dino with grounded pre-training for open-set object detection. arXiv preprint arXivv2303.055499 2023d. 3
K., Wu, X., Liu, H., Malladi, S., et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406. 18521, 2024. 1,
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- -f-tuugghtth prompting elicits reasoning in large language models. Advances in neural information processing systems, 01-24824-24877 2022.
Liu, Y., Li, Z., Huang, M., Yang. B., Yu, W., Li, C., Yin, X., lin Liu, C., Jin, L., and Bai, X. Ocrbench: On the hidden mystery of ocr in large multimodal models, 2024b. URL
https://arxiv. org/abs/2305 07895. 2
Masry, A., Long, D. X., Tan, J. Q. Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244 2022. 1, 3, 4, 5, 8,
Yan, A., Yang, Z., Wu, J., Zhu, W., Yang, J., Li, L., Lin, K., Wang, J., McAuley, J., Gao, J., et al. List items one by one: A new data source and learning paradigm for multimodal llms. arXiv preprint arXiv:2404. 16375, 2024.
3
Masry, A., Kavehzadeh, P., Do, X. I.,, Hoque, E., and Joty, S. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint PrXiv:2305.14611. 2023. 1.2
Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11441, 2023. 7,7
OpenAl. Gpt-4 technical report, 2023, 6
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., and Zhao, H. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 3
Pasupat, P. and Liang, P. Compositional semantic parsing on semi-structured tables. arXiv preprint 1508:05000.00000 2015. 4
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis. G., and Smola, A. Multimodal Chain-of-thoughe reasoning in language models. arXiv preprint arXiv:2302.0092. 2023.
Rizzolatti, G., Riggio, L., Sheliga, B. M., et al. Space and selective attention. Attention and performance XV, 15: 231-265, 1994.
10

---

REFOCUS
Zhu, F,, Lei, W. Huang, Y., Wang, C., Zhang, S., Lv, J. Feng, F., and Chua, T.-S. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021.

---

REFOCUS
A. Overview
In the supplemental materials, Appendix B contains additional details for REFOCUS, including editing tools($B.3), and experiment configurations such baseline model setups ($B.4). Appendix D discusses supervised fine-tuning details including experiment setups($D.2), dataset statistics($D.I) and result analyses($D.3), Appendix C demonstrates prompts in REFOCUS. Appendix E shows qualitative SFT output examples.
B. REFOCUS Prompting Details
B.1. Additional Example
Which aircraft did not originate in the USA ?
Aacratt
Rokk
Commande
Light
2i wit wing Fomm S018
10:00
coraliee Mine Foe
Beechorah
Lightrrrspan
Cesara Canuse
Dito rid Titn
U/54
Cimor Erars
Lugh Huunpeet
2 snavtc
Costro A2CC Goklen Caple
Lightcarapan
2000
Cerara 421C Gullen Eagle
2216
Light turspot
po
GPT-4o: Partenavia P.68 Observer
oo
GPT-4o: Partenvia P.68 Observer
Figure 7. REFOcUS unleashes better OCR for GPT-4. In this example from TableVQA (Kim al., 2024), REFOCUS GPT-4 conducts the edit action highlight_colum. With this simple action, GPT-4 can focus more on the important subarea, and recognize the characters better.
B.2. Additional Analysis
Table
Chart Horizontal Bar
Model
VWTQ
vwTQ_syn
VTabFact
CharXiv
Vertical Bar
GPT-4o Model
Text input
68.1
69.6
80.0
\
66.2
74.9
Figure Input
61.0
68.4
88.4
47.9
75.2
74.9
Text Figure Input
67.5
72.8
91.6
75.7
81.7
GPT-4o REFOcUS
Figure Input
77.2
82.8
90.8
57.3
85.4
81.2
+16.2
+14.4
+2.4
+10.6
+9.7
+0.5
Table 5. REFOCUS empowers GPTT-4o to achieve the performance as if given gold text input. The text input are mainly esv tables, as detailed Section 3.1. Notice that Table reports the conversational performance without visual editing, whereas the performance discussed here is based on direct question answering, leading to minor differences.
B.3. Visual Editing Tools for Charts
As introducted in Section 3.2, we adopt three types of visual editing method in REFOCUS: mask out, draw box. and highlight color using Python code. In our experiments for chart problems, we deploy list of tools as follows:
Highlight Bar at X, which overlays a light red color over a bar in a vertical bar figure that needs to be focused. It operates based on the x-value of the chosen bar, as in Figure
Highlight Bar at Y, which overlays a light red color over bar in a horizontal bar figure that needs to be focused. It operates based on the vavule of the chosen bar, in Figure
Mask Bar at X, which places a white mask over bars in vertical bar figure that do not need attention.
12

---

REFOCUS
Mask Bur at Y, which places white mask bars in horizontal bar figure that do not need attention.
Draw Bar X, which overlays a solid red bounding box over bars in vertical bar figure that need be focused on.
Draw Bar which overlays solid red bounding box over bars in horizontal bar figure that need to be focused
B.4. Experiment Details
For all the experiments, the temperature is set to O. All evaluations are done using GPT-4, where it's given prediction and the gold answer, and decides whether the prediction correct or not. For the Section 4 experiments, we use NVIDIA Quadro RTX 8000 GPUs for the inference of open-source multimodal LLMs, including LLa LLaVA-NeXT-7B, LLaVA-NeXT-13B LLaVA- NeX-3-B. and Phi 3 vision (4B). These experiments cost around 40 GPU hours.
13

---

REFOCUS
C. Prompts
We show REFOCUS prompting details for table problems and chart problems with running examples randomly selected from VWTQ and Horizontal Bar datasets as follows. Prompts for Table Problems:
SYSTEM PROMPT
You are a helpful multimodal AI assistant. [MORE INSTRUCTIONS.... ...]
For each turn, you should first do a "THOUGHT", based on the images and text you see. If you think you get the answer to the intial user request, you can reply with "ANSWER: <your answer>" and ends with "TERMINATE".
Initial Prompt Request
some tools
help you. are python godes. They are tools. py and will imported you.
You will be given tble fiure: image,1 I and a question.
Notice that
an At assistant are not good at anvwering questions when there are too many unnecesszry and inrelvvnn information. You should detertine which are the relevant python list. You should the colama headers.
You should also desermine specify
columns the question.
the relevant rows You could solect the tools to tocus on some cotumas / rows, or mask out souse columns/ / rows. Use whichever tool you think is more upprogziane. Helow are the tools in tooks py: python
question, specify
a python list You should the given now headers.
focue_on, sosur_on_couumee columno with highlighttimage columns
all columos bownding boses):
This function is useful whes you want to fooms en some specific columns the image.
10
It does this by adding light mansparent red highlight the columns thit need to be focused on.
11
For example, can focus columns tible that relevant your analysis.
12
Returs drawed image.
13
14
Args:
15
image (PIL. Image Image): input image column names focus
16
colmatss to music (List] str D):
17
all columns hounding houes (Diet ID: a dictionnry of hounding boses for all colomns in the image. key is calomn some and value is the hounding box of thit column. Fach bounding box is in the format ('x1"; xl, "yth yi, '22 x2, 'yan y2].
18
19
Retures:
20
image with with_ffcuuseeett focused columes (PII. Image Imsgee:: the image with spacifled columns focused on
21
22
Example:
23
image Imsage opeeo.ppe"""aa "suapte_ img iog")
24
image with focused columns = focus, on columns, with Nighlightümage, ["Year", "Name"]. L'Yeur: UKL's 0.1, 1yts 0.1, 'x25 0.3, 'y2" 0.91. 'Temm' [XX1 04, "yt O.1.
*23 0.6 y22 099, "Name": ('x1% 5 07, "yI": 0.1, 'x2%, 0.9. 'y2) (.9))>
25
display (image_ with focusedd columas)
26
27
28
def fovas
with highlight (impge,
focue alll nows, bounding buses):
30
function useful when you want to foens some sposiffi rows of the inage.
31
It does this by adding light transparent red highlight to the nows that need to be focused on.
32
For exmmple can focus the rows table that are relevant to anilysis.
33
Retur drawed image.
34
35
Args
36
image Image Imagey: the inpux image
37
focus, (Liss str list noss bounding boxes (Diei [iiet) 1 a dictionary bounding boxes, all nows image. key is row header and value is the bounding that row. Fach bounding
38
box is in the format I' i xl. "yl% y1. 'x25, x2, 'y2': y2].
39
40
41
image, with focused rows (PIL. Image Image' the image with specified rows focused on
42
43
Example:
image Imuge opent 'sample img ips')
45
image with. focused rows = focus, on nows, with hightightimmpp. ['1972"]. ["Your's ( 0.1, 1915 0.1, 'x2% 09. 192% 0.15), 11969% (XFS 0.1, 1915 0.2, 1x2% 0.9. >2% 0.5). 11972% I' 0.1. "yl': 0.6, 'x2: 0.9, 'y2's 093
16
display (image, with focused_rows)
47
48
49
def focus, on columne with maskimmgge columns, tu focus on, all weauuoool__.boeoo bounding bosesi:
50
function is useful when you want to focus some specific columns image.
52
It does this by masking out the colums that are not needed.
53
For example, you can focus the columns table that are relevant your analysis and ignore the rest.
54
Retum the masked image.
55
56
Args:
57
image Image Image': the inpux image colurnes mark (List] str list of column all_cotunns_ bounding boxes [Dict] Diet ID: dictiontry of bounding boxes columns image, key column name value is the bounding that bounding box is in the formal ('xl': xl. 111 yl. 'x2% x2. 1y2%, y2l.
58
focus
60
61
Returas:
62
image, wwthh/hhhwwuaeeeeee focused columns (PIL Image Imager: image with specified columns focused on
63
64
Example:
65
image Imuge 'sample_ ipx')
image with_focusee columns focos_on, on cotumnscimage. "Year" "Name"), ("Yoart EXI' 0.3, 1yt' 0.1, 1x2% 0.3, 'y25 0.91, "Team": ('x1) 0.4, "y1's 0.1. 'x2% 0.6. 'y2
0.9). "Nume": ('xl: 0.7. 'y1', 0.1, 'x2% 09, 'y2': 0.91))
67
display (image with focused_ columns)
68
14

---

REFOCUS
def mecus_on_rtos_seee wilh, miskimm twwe_tt__oooeeeoeo to too__to_fcooscoo wew.ww_bouddngg boxest:
This function is useful when you want to focus on some specific rows of the image. It does this by masking out the ruws that are not needed. For example, can focus Return masked image.
table that are relevant your analysis and iguare rest.
Args:
10
image (PlL., Imagel: the inpus image rown focts on (List str DD: list headers focus
11
12
alL LLLLnww_Dounding boxes (Diet) Diet a dietsouary of bounding boxes for all rows in the image, key is row header and value is the bounding box of that row. Each bounding box is in the format I's1' xl, "y1h yt. 'x2 x2, 'y2' y2).
13
14
Returns:
15
image, with focused, yows, (PII. Image Imagey: the image with specifled rows focused on
16
17
Exaaple:
18
image Imsage opent "sumple, lig.ing"" ing")
19
image fochufoccsedd_..oo yows - focus, onnrosstimgge ('19027), ["Yeart ( 0.1. 'ya O.I, "x2% 0.. y22 0.15), 119697 ('X1 0.1, "y1s 0.2 'x2% 0.9, 'ya: 0.5). 19922" ['xl: : U.I. "yI"; U.S. 'x2;; 0.9, 'y2: 0.91)
20
display (image, with sows))
21
22
23
def focus on columnne_ with drwwcimage, columni to focus on, all colames bounding boxes):
24
25
This function is useful when you want to foens os some specific calumns image.
26
It does this by drmwing red box around the calumns that need to he focused on.
27
For example, you can focus the columns tbble that are relevant your analysis...
28
Retur drawed image.
29
30
Args
31
image (PIT. Image Image): the inpur image (List)
32
list column columns, bounding houes (Diet ID: dictionary of bounding boxes for all colomns the imago, key is column nome and value is the bounding box that column. Each bounding box is in the fommut ('x1% x1, 'yt% yl, 'x2 x2, 'yan, y2l.
33
34
35
Retures:
36
image, with focused colomes (PII. Image Image); image with spaciffed columns focused on
37
38
Example:
39
image- Image opent" "sampple img.ine"
40
image with focused, columns = focus, on pelommstimage, (" (YYer", "Name"]. Y:eur": PX11 0.I, SS1 0.1, 'x25 0.3, 'y2% 0,91. "Temm": PXXl 044, 1y1' O.I. 'x2's 0.6. 'y2 0.9). "Name": I'xl': 0.7, "y1s O.I, 'x2% 0.9, 'y2": 0.91])
41
display (image, with Socused podumis))
42
43
44
def focus on_
www_sonn__sws_o_o_o www.rrowwllll.. with deahsdddssiiaga cows, Jo foems on, all_ fowss hounding. hores);
45
46
This function is usaful when you
foem
specific
the image.
47
It does this by drawing a red box around the nows that need to be focused on.
48
For example you can fucus on the rows in a table that are relevant to your anilysis.
49
Retur the drawed image.
50
51
Args
52
image (PII. Image Imagey:: the input image
53
focus (Liss [ list beaders focus : dictionary bounding boses all nows image. key hender and value the bounding that now. Each bounding box is the format 1'x1%; xl. "y1' y1. 'x2% x2, '225 y21.
54
nows bounding
55
56
Returns:
57
image, wwthwwhhhwuuueeeee focused Jows Image Imagek the image with specified rows focused on
58
59
Example:
60
image Image opend 'sample imm.
image with pows focus coluume with hightightttnnage (119727), ("Year": I'sl O.I. 1y O.I, 'x2's 0.9 '52' 0.15), 11969% U'U1 0.1, 'yU's 'x2's 0.9. y2" 0.5). "1972% 1 0.1. 'yl't 0.6, 'x25 0.9. "y?". (.9))
62
display (image with rocused_rows)
63
65
a GOAL Dased un the above tools. I want you to reason about how to solve the USER REQUEST 4 and generate the actions step by step (each action is a python jupster notebook code block) to sulve the requst.
66
You may need the tools above process the images and make decisions based visual oupputs of the previous code blocks.
perfect, you should use these wols asdist you in reasoming The iepyer notebook has already executed the following code to import the mecessary packages:
ability
images.
68
69
python
70
from PIL import Image
71
IPython. display import display
72
from tools impoct
73
14
15 REQUIREMENTS
76 The generated actions can resulve the given uset request # USER REQUEST A perfectly. The user request is reusumable and cun be solved. Try your best to solve the roquest.
77
2. The arguments of tool must be the same format spesified in TOOL LIST ft.
78
I you think you got the
Please extract the final intial user request are stoted PiL. Image objects named image image_2. ... imaaeee n.. You can use these images your code blocks. Use display () function
use ANSWER:
in ANSWER. <tinal
and ends with TERMINATE.
79
images
to show the image in the notebook for you too see.
80 5. Use few tools as possible. Only use the tools for the use cases written in the tool description You can use multiple tools in single action.
If you have multipl
please separate them image, most like you need look multiple columas instadd single one.
marks. example,
'Alice' and 'Bob', you should
Alice Bobb'
82 When you focus on columas
83 8. If you do not think you have enough informatio to znswer the question on the images returned by the tools, you should directly answer the question bused on the original image.
84 Helow are some examples how use the tools to salve the user requests. You can refer them for help. You can also refer to the tool descriptions for more information.
15

---

REFOCUS
Four in-coment examples here]
USER
Golfer
Country
Play
Championship
Champions
3: 2003, 2004, 2008
7: 1999, 2002, 2003, 2005, 2006, 2007, 2013
1999, 2000, 2001,
Woods, TigerTiger Woods
United States
18
2005,
2006, 2007, 2009, 2013
Ogihy,, GeoffGeoff Ogilvy
2008,
Clarke, DarrenDarren
Northern
1: 2000
2003
Clarke
Ireland
Els, ErnieEnnie Els
South Alrica
2: 2004, 2010
Mahan,
Mahan
United States
1:
2:10
Mickelson, PhilPill Mickelson
United States
1: 2009
1: 2008
England
me the
à USER Rounding Box Info: columes, bbox, where keys column headers, and values ane column hounding hoses, nows Noox, where keys row headers and values are row hounding boxes. The "Chanpionship', Invittional "Champions]] "Malum Hunterfounter Maltar, Mickeloo,, PhilPhil Mickelson", ; "Puulee, linlan Poulter
colomes the image Golfer', "Countyy' 'Wins', Wondk, 'Ogilyy, GeofTGeoff Ogilvy", 'Clarke, DarrenDmmen Clarke "El,, EmixEmii
Golffe"" "Woods, TigerTiger
8 USER IMAGE stored image__ de PIL. image:
Now please penerate THOUGHT ACTION TERMINATE:
neaded, reply ANSWER cyour
FINAL ANSWER: <himal
RESULT
Thought + Action 0
THOUGHT O: To find the number of wins His had, seed to focus on the "Wins" column and the row for 'Es, Braistimie Bis".
ACTION
python
image_wit www.wl/fouuueeeoo.oooooo
columms columns focus_on, cacosooo_oooaaee columes with, dwwwimage_e J. "WWnns I, columns_bbor
image with focused mou - facus, rows, with devwrinage with, focused, pollanes ("Els, FrnicEmic Bis") roox_
Observation 0
Execution success. The output as follows.
Golfer
Country
Wins
Match Play
Championship
Invitational
Champions
Woods, TigerTigen Woods
United States
18
2004,
2002, 2005, 2006, 2007,
2005,
2000,
2008
2013
2006, 2007, 2009, 2013
Ogilvy, GeoffGeotf Ogilvy
Australia
2006, 2009
2008
Clarke
DerrnnDarien
Ireland
2
2:0000
1:0003
IC mma--mi Eis
Sor Afria
2014
Mahan, HunterHHutter
United States
2012
2020
Mahan
Mickelson, PhilPhi Mickelson
United States
2
1009
1: 2009
Poulter, lanian Poulter
England
2:10010
2012
Answer the question tell me the number of wins els had. You can turn the table image into text and answer with step of thinking.
Reply with ANSWER: <your answer>
Please extract the final answer in FINAL ANSWER: <final answers and ends with TERMINATE.
Thought + Action
ANSWER: The number of wins Els had FINAL ANSWER: TERMINATE
16

---

REFOCUS
Prompts for Chart Problems:
SYSTEM PROMPT- CHARTS
You are a helpful multimodal AI assistant. [MORE INSTRUCTIONS... ...]
For each turn, you should first do "THOUGHT", based on the images and text you see. If you think you get the answer to the intial user request, you can reply with "ANSWER: <your answer>" and ends with "TERMINATE".
Initial Prompt- Request
tools
All python codes.
toels py and will imported for you. will gaven figure: image__ and question.
Notice you, an assistant are good at answering questions when there are too many unnevessary inreevant information.
you are dealing with a vertical bar chat figure. you should determise which wr the relevant x values to the questio, and specity them in a python list. You should use the gaven x value numes.
dealing with horizontal bar chart ligure. you should also devermine which the relevant values the question, and specify them python list You should the gives y value names.
Helow are the tools in tools py:
python
def focur_on__ ve_ut_oo__vattteecc. with mastrimage, Laaless_ Je focus_on. aلl_X_ values_ hwwnding___daooooo hoxes):
This function is useful whes you want to Focms on some specific x villies in the image.
10
It does this by masking out the values thit are not needed.
11
This function is especially useful for vertical bar charts.
12
For example you can focus
x values chart that relevent to your anclysis ignore the rest
13
Retura the masked image.
14
15
Args
16
image Image Image); input image x values, tn focus on (1is (tr D:: liet of x values to focms on.
17
18
all x. Vallues hounding bones (Dier Dier dietionnry of bounding boxes for all valies in the image. key is value and value is the hocading box of that value. Each format ('xt' xl, "t" 'x2" x2, 'ya y2).
bounding
19
20
Returas:
21
image, with Tocused x values (PIL. Image. the image with specilied values focused
22
23
Example:
24
image Imuge opent "sample ing.ine")
25
image with focused x values focus valaestimage, (12006", "2006"), ("2005% PXTT O.I, yIU 0.1, 1x27 0.3, (y2) 0.91. "2006"; (SS3 0.4, "y15 0.1, 122 0.. y2: 09). "200"" I'xl' 0.7, 'yt't 'x2t D.,, 'y25 (9]]))
26
display (image with focused x. values)
27
28
29
def focus on values mask<image,, y. values focss on, y. values. bounding, buses):
30
31
This function is useful when you wamt to focus on some specific y valloes in the imuge.
32
It does this by masking out the values that are not nceded.
33
especially useful For cumple you can focus un the y values in chart that are relevant to your amilysis and ignore the rest.
This function
horizontal
34
35
Retum the masked image.
36
Args
38
image (PIT. Image Imagey: input image
39
y values to focus_ on (Lisx str DJ: list of y values to focus on.
40
values hounding boves (Diet (Diet dictionary of bounding boxes for all valuex the imuge. ley is value and value the bounding box of that value. Each format ('xl' xl, y1 "x2' y2a y2].
bounding
41
42
Returns:
47
image with focused y valuos (PIT. Enage): the inage with specified values focnsed on
45
Example:
16
image= Image opent "sample img ipg')
47
image with Jocused y values focus valoestimape, ['0', "10'1. 1'0% PsU's O.L. SUS O.L. 'x2 0.9 'y2% 0.15), '10": ('x1's 0.1, 'yl' 0.2, 'x25 0.9 '22 0.5),
"20": ('xl' yys 0.6, "2' 1y2% 0.91))
48
49
50
def focus valuoes with deaweimuge, x values to focms on. all x values bounding boxes):
52
This function is useful when you want to focus on some specific a valuoss in the image.
53
It does this by drawing red box around the values that need be focused on.
54
This Junction is esperially useful for verticul bar charts.
55
example, you can Retum the masked image.
values chart
relevant your anilysis.
56
57
58
Args
59
image (PIL.. Image Imagel: input image x_ v_values__a to Socus_oom on (List] str Di list of x values to focus cc.
60
61
alL values, hounding hones (Diet) Diet (): dictionayy of bounding baxes for all values the image. key is value and value the bounding box of that value. Each
bounding
the format ('x1' xl. yy'' 'x2% x2. 'y2' y2).
62
63
Keturas:
64
image_ wwthhfoccuse_otppoooooooom x values Image Imago): the image with specified values focused on
65
66
Example:
67
image image, opent "sumplee_ing ips')ps")
68
image with focused X values focus_on_ vinaaessimage, (12006", "2006"), ("2005%: PXTN O.,. "yU O.J. 'x2% 0.3, (y27 0.91, "2006% ('xI' S 0.4, "y15 0.1, 'x2% 0.. y2% 09). "200"" I'xl' 0.7. 0.1. 'x27 09, "y25 0.91)
display (image with Louusd___vlless
70
17

---

REFOCUS
def focus_on____ v_luv__vlleewiiee with drawtimgge y_ y_ oo, Villy_value__buu bounddigg boxes):
This function is useful when you want to focus on some specific y values image. It does this by drawing a red box around the y values that need to be focused on. This function is especially useful for horioontal bar chans..
For example,
focus y values chart Retur masked image.
relevant your andlysis.
Args:
10
image Image Imaget: the input image y. y_valuesu to oo [Lise] sn: D:
11
of y values focus
12
alL_yvvluee bounding hoves (Dicct) Diet D): dicnceary of bounding boxes for all valkes in the image, key is value and value is the bocadding box of that value. Each bounding bot is in the format ('xl'; : xl. "yt yl. 'x2% x2, y22 y2l.
13
14
Returns:
15
image, with with focused y. values (PIL. Imag.. Image)) the inage with specified values foensed on
16
17
Example:
18
image Imsage opent "sumple img.ing ing")
19
image with_focusee y. values focus_onn, onn.s.onn... Baros_on__votteiineeeeo vuteestimage, I'D", "10%). (10% PXLS O.1. yyt D.I. 'x22 0.9. 'y2: 0.15). '10": (XE2 0.1, yyt: 0.2 'x2" 0%. 'y2" 0.5),
220: ('xl: 0.1. yy: 0.6 "x2' 09, '22% 0.91))
20
21
22
def jacus_on a. with highlight limage, x_ x_values__o to foc_soss____... allx_values BiL___alle__bouddnge bounding_
23
24
This function is useful when you
focus
spocific vilues image.
25
this by adding light traneparent red highlight
values that need to be fecssed on.
26
This function is expecially useful for vertiall bar charts...
27
For example, you can focus the x values in chart that are relevant to your analysis.
28
Retur masked image.
29
30
Args:
31
image (PIT. Image Imagey: the inpur image fo_fuuus_oo (List] 1:: of values focus
32
33
x values hounding bones (Diet (Diot D: dictionary of bounding boxes for valuess the image, Tey is vallee and value the bounding box of thut value, Each bounding box is in the fonsat ('x12 xl, "yl yt, "22% x2, y2a y2|.
34
35
36
image with focusod x values (PII. Inagge- the image with specified values foeused on
37
38
Example:
39
image- Image opent "sumple, ing"
40
image with fucused x values focus, on x, valles(imaps, ('2006". "2006"), 122005%; 1'15 O.I. 151 0.1, 1x2% 0.3, 'y2' 0.91. "2006"; ('El 044 "y15 0.1, 'x2% 0.6. 22: 0.9). 2000% I'x1': 0.7, 'yt% 0.1, 's2' 0.9, 'y2' 0.91))
41
display (image, with focused_ x x_valuss)
42
43
44
def facus, valee with highlight (image, y_values_ so focus_on, all_y_.vlues Sa___aalee__ooudddee bounding hoxces):
45
46
This function is usafal whes you
foems
specific y villees image.
47
It does this by adding light transparent red highlight tu the y values that need to be focused on.
48
This function is especially useful for horioonal bur churts..
49
For example. you can focus the y valuos in clart thee relevent your anilysia..
50
Retum masked image.
51
52
Ares
53
image Image Imageh: the imput imagee focus (List str D:: of y values focus y vllaevalees_ hounding boves (Diet) D:: dictionary of bounding boxes for all y valkes the image, key is value and value is the bounding box of that value. Each
54
55
bounding box is in the furmat ('xl's xl, "y' yl. "x2% x2. 'y2': y2].
56
Returns:
58
image, with focused, y values (PIL. Imtge Imuge. the imuge with specified y values focused on
59
60
Example:
image: Image opent" 'sample img ips')
62
image with, focused y values focus on y vilaes ('0', "10'). 170% PXF5 0.1, 'yt's 0.1, 'x2' 0.. 1y2% 0.15), "10": (1E55 0.1, 'yl' 0.2, 1x25 0.9 'y2h 0.5).
"20": ('x1: 0.1. yy1 0.6, "x2% 0.9, 122 0.91))
63
65 # GOAL Dased un the above tools. I want you to reason about how to solve the USER REQUEST 4 and generate the actions step by step (each action is a python jupster notebook code block) to sulve the requst.
66
You may need the tools above process the images and make decisions based visual oupputs of the previous code blocks.
perfect, you should use these wols asdis you in reasoming The hepyer notebook has already executed the following code to import the mecessary packages:
ability
images.
68
69
python
70
from PIL import Image
71
IPython. display import display
72
from tools impoct
73
14
15 REQUIREMENTS
76 The generated actions can resulve the given uset request # USER REQUEST A perfectly. The user request is ressumable and cun be solved. Try your best to solve the roquest.
77
2. The arguments of tool must be the same format spesified in TOOL LIST ft.
78
I you think you got the
Please extract the final intial user request are stored PiL. Image objects named image image_2, ... imageen nn. You can use these images your code blocks. Use display () function
use ANSWER:
in ANSWWANSWEL <tinal
and ends with TERMINATE.
79
images
to show the image in the notebook for you too see.
80 5. Use few tools as possible. Only use the tools for the use cases written in the tool description. You can use multiple tools in single action.
81 If you
you have enough information 82 If all x values are relevant
the question question for vertical bur chart, you do not need to focus on my specific
images ceturned the tnols, you should direttly answer the question based the original image. You should directly unswer question based
the uriginal image.
83 8. It all the values are relevant to the question for hocizcntal bar chart, you do not need to foess
specific y values. You should directly answer the question based on
the original image.
84 Relow are some examples
use the rooks to solve user requests. You refer them for help. You can also refor the descriptions for more information.
18

---

REFOCUS
# Four in-coniest examples here]
REQUEST
cordtrr Invormorn stletro
1BC
Arraing
Mest
Hew sume
Mori
Collagoe stotiats
What is the average of blue bar above 20 value?
y_salues, storing y intervals, Mbux.. storing intervals and coundimsess The x values image ane: [). The values in the image are: ("Lending/Iveees standands KKepping information and money safe'. "Koowhow own industry workings: "Meral standards "Cost of eapinal "Generating social retums on investment Gemensingg financial
Bounding
saluues storing
x_intervals, bbox. storing
retums on Accessibility
'Speed' Flexibility "Willinnees take risks
intervals the image (). They intervals the
image, in image.
Now please
THOUGHT Oud ACTION in RESULT. To action needed, also with ANSWER
Please extract the final FINAL ANSWER: <linal
à RESULT
Thought + Action 0
THOUGHT O: This is honzonial bar chart image. need to focus on the values where the blue bar is above 20 and calculate the average of those values.
ACTION D:
python
image wwwll_fooseee y._ values focus_on, only. www_yaalles_____eeeaaa_a____eeea driih_deaw/immge.ll Accessibility "Speod". Flexibility
"Willingness to take risks 'J. bbox)
display (image_ with wwwhsfoccood_........... valuues)
Observation 0
Execution success, The output is as follows.
Lording
Keyping onte vanten meney
X
Censainn intur invooeee
Survtidly
3M
Beien About More
Collagne cunlete
Don
19

---

REFOCUS
Answer the question What is the average of blue bar above 20 value? You can turn the table image into text and answer with step of thinking. Reply with ANSWER: <your answer> Please extract the final answer in FINAL ANSWER: <final answers and ends with TERMINATE.
Thought + Action
ANSWER: The blue bars above 20 are for "Accessibility" (21%), "Speed" (23%), "Flexibility" (26%), and "Willingness to take risks" (29%). To find the average:
21 + 23 26 + 29 99
Average-
24.75
FINAL ANSWER: 24.75. TERMINATE
20

---

REFOCUS
D. Finetune Details
D.I. REFOCUS Dataset Statistics
The detailed dataset statistics are demonstrated in Table 6. One example input data can viewed in Listing
Horizontal Bar
Vertical Bar
Total
QA Data (Masry et al., 2022)
4,990
10,069
15,059
REFOCUS Data
4,722
9,622
14,344
w/ Editing
4,220
8,599
12,819
Table 6. Detailed statisties about REFOCUS Data. We count the total number of our training cases, and the ones with visual editing.
D.2. Finetune Experiment Details
For the fine-tuning experiments in Section 5, we use 8 NVIDIA RTX A6000 GPUs (48GB RAM per GPU), following the Phi-3.5-vision github instructions". We adopt full training, and iterate through the hyperparameter searching space to find the best performing set to report as our result score. The hyperparameter search focuses specifically on (1)learning rate, (2) epoch number, (3) whether to include edited image in training input, with learning rate ranging (Se-5 5e-6 5e-4 le-4 le-5 le-6 5c-7), epoch ranging in (1 2), and all other parameters default values. In Table present detailed hyper-parameter configurations for results in Table 4.
w/ default QA Data
REFOCUs VCoT
w/ REFOCUS CoT
data type batch size
bf16
bf16
bf16
64
64
64
learning rate
5 x 10-7
x 10-6
5 x 10-6
epoch number include edited image in input
2
2
2 Yes
No
No
Table 7. Hlyper-paramette settings for our best fine-tuned models.
D.3. SFT Result Analyses
Looking at the SFT results in Table 4, we see a consistent improvement on the chart problems using REFOCUS data. If we use the de facto QA data to finetune, the chain thought ability will be impaired, comparing to the original model. Also, the improvement on Vertical Bar problems reaches 5.4%, which quite large for 14k training data.
E. SFT Qualitative Examples
In this section, we show some example output from the Phi-3.5-vision model finetuned on REFOCUS visual CoT data in Figure 8. Notice that the model output are in text formats with bounding box coordinates for the areas to focus on. We draw these areas in red boxes for illustration purposes. The examples are selected from the Horizontal Bar set and Vertical Bar set.
21

---

REFOCUS
Listing 1 Example REFOCUS data input for SFT experiment. The "responseO" and "responsel" are GPT-4o outputs that generate codes and answer based on the edited image respectively. We use vycot_input as the target answer.
(
"id":
"query": "As of 2021, how many championship titles had Ferrari won?",
"answer": "16",
5
"source": "h_bar",
"images": ("two__co__00352..pg""
"responsed": "This is a horizontal bar chart image. I need to focus on the part where the y-axis value "Ferrari" find out how many championship titles they have won.", "edi ted ted_teege_m
]ng" "response1": "As of 2021, Ferrari had won 16 championship titles.", "focus_areas":
"x1": 5,
14
"y1": 38,
"x2": 795,
"y2": 72
}1, "vcot_input"; "This is horizontal bar chart image. I need to focus the part where the y-axis value is "Ferrari" to find out how many championship titles they have won. The areas to focus on in the image have bounding box coordinates: [["x1": 5, "y1":
20
21
38, "x2": 795, "y2": 72)]. Looking at these areas, As of 2021, Ferrari had won 16 championship titles."
22
22

---

REFOCUS
Q: What's the percentage value of "A great deal" opinion in Indonesia A: 27
Majerities believe rights organtactions inflaence way thinga are going their countries Miric rophoapeerorro presuing..... tyfocaane Aiige inggrny eningg
Majerities haman rights organizztions
Majoritis rights arganizations
uance the way thinge are geins their countriee monee www.ee/epeeeeeee.eeeeeooo af the siy arepaiag Iteal audlyy
influeece steing in their coentrics
thounre weteentttteeaaee tte offectunce fltt aigplta WE3t?
Sunns 2007 Saluttu- Von www.uuuarr.ooo Au M4
eou marerssanoi otvers
now nta nonenume
ReFocus GPPT-4 Prompting Result
Phi-3.5-vision finetuned with ReFocus Result
Q: What is the value of Cattle bar ?
A: 40.7
Shar of trepiaal Seferacttiion agplechrrall prodiats aua
Share to an deforaatee frces sgnoditee prodects
ReFocus GPT-44 Prompting Result
Phi-3.5-vision finetuned with ReFocus Result
Q: What's the highest value among the blue bar ?
A: 41
Editing Needed
ReFocus
GPT-44 Prompting Result
vhisson.v vision finetuned with ReFocus Result
In which year M&S had highest gross profit margin 7
2017
You
Focus
GP Prompting Result
Phi-3.5-vision finetuned with ReFocus Result
Figure Phi-3.5-vision finetuned with REFOCUS visual chain of thought data oulputs the areas to focus on. For illustration purposes, we draw these areas in red boxes, and compare with the REFOCUS GPT-4o prompting output.
23


Reviews:
[{"review_id": "uGMcJfQ8rS", "reviewer": "Reviewer_cQeo", "summary": "The paper proposed a LMM + tool-using style framework for structured image understanding. Specifically, the framework provides tools to edit the original image and either highlight the important region or remove irrelavent region. Results show that this framework generally improves LMMs zero-shot performance on understanding tables, charts, etc. The paper additionally show that using the framework to curate SFT data provides better supervision signal for these tasks.\n\n## Update after rebuttal\nThe rebuttal provides additional infomation, I am keeping my already positive evaluation", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback. Below, we provide a detailed response to address the concerns and questions raised:\n\n* Q1. Tool Errors\n\nWe carefully examined this phenomena through manual checking, on around 40 data cases each for Table VWTQ and Chart Horizontal Bar datasets. \nFor table editing, most of them (roughly 80%) are correct. For the problematic ones, there are two big error types. The main error (\\~80% of which) is that sometimes, both column editing and row editing are needed, but only one of them was applied, making the original problem, e.g. counting specific rows, a bit easier but not easy enough for the model to solve. A small portion (\\~20% of which) is that it could mask out important information (e.g. some rows or columns)  that should not be masked out.\nFor chart editing, it’s similar that most of them(\\~90%) are correct. The main error is that sometimes the python program is incorrect, and the wrong area gets edited, e.g. the chart should focus on the last three bars, but the last 4th instead of the last 3rd bar was highlighted, therefore the final calculation uses the incorrect number of the bar, etc.\n\n* Q2. Remaining Errors if Visual Editing is Accurate\n\nSome complicated reasoning cases still remain a challenge. For instance, if a chart image has three bars, and all of them needs to be considered, compared, and calculated with each other, then even if ReFocus highlights all three bars, it still makes the same mistake as if no edit is performed. \n\n* Q3. Performance Difference.\n\nFor VTabFact in Table2, we believe it is because Phi-3-vision may not deal with visual prompts that well, compared to dealing with the original image, because it was largely trained on OCR tasks and has seen many original text-rich images. We believe that the programing ability of the multimodal LLM and the visual prompt understanding ability will both affect the effectiveness of ReFocus.", "claims_and_evidence": "The claims are well-scoped and the results supports the claims.\n\nAdditional explanation/analysis are needed; see the Experimental Designs Or Analyses section.", "methods_and_evaluation_criteria": "The evaluation setting makes sense; The strength is the paper also includes finetuning experiments;", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are mostly solid and informative;\n\nAdditional anaysis is needed in:\n- It's good to show error analysis on the tools and how it will be propagated to the final performance; \n- Categorize what are the remaining errors that cannot be addressed even with accurate visual editing;\n- For some categories, ReFocus fail to yield better performance, e.g., VTabFact in Table2; What's the reason? Additional analysis would be helpful", "relation_to_broader_scientific_literature": "This is in line with research in multimodal reasoning on abstract visual concepts.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is clearly written and easy to follow", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYSsBEsPoL", "reviewer": "Reviewer_phJH", "summary": "This paper focuses on the structured image tasks, using the Python codes to edit the input image as the \"visual thought\" and the modified image can be more focused on the target information of the question, which benefits the accuracy of the answer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. Below, we provide a detailed response to address the concerns and questions raised:\n\n\n* Q1. Evaluation Dataset Selection.\n\nThe major reason for testing on ChartQA instead of on DVQA is that DVQA is a synthetic (chart and data) dataset with template questions, while ChartQA and CharXiv include real charts and data with open-form questions that have been manually checked. Additionally, ChartQA has been much more widely tested by many recent models (as in Gemini and GPT reports) than DVQA so that it’s easier to compare with them, and CharXiv has its unique challenges of scientific chart reading.\n\n\n* Q2. Evaluation Set Size Concerns\n\nWe want to emphasize that small scale evaluation data does not mean unreliable evaluations. For example, datasets like Winoground (CVPR 2022), which contains only 400 examples, have significantly influenced VLM advancements. Similarly, although focused on different aspects, other widely adopted benchmarks—including WHOOPS! (ICCV 2023), LlaVA-Bench (NeurIPS 2023), Visit-Bench (NeurIPS 2024), ConTextual (ICML 2024), VibeEval, and Visual Riddles (NeurIPS 2024)—comprise 90, 500, 576, 500, 269, and 400 examples, respectively, and have been pivotal for evaluating VLMs. Our total evaluation set has a size of 2219 instances, and should reflect a reliable evaluation result. \n\n\n* Q3. Performance Fluctuations between different models. \n\nYes, there are performance differences between different multimodal LLMs, but we believe the Table1 results show relatively consistent improvement. The programing ability and the visual prompt understanding ability of LLMs will both affect the effectiveness of ReFocus. However, we want to emphasize that (1) ReFocus is a simple yet effective general pipeline that can work with different multimodal LLMs, and (2) we add the fine-tuning experiments to reduce the reliance on prompting to a certain degree. \n\n\n* Q4.  Limitations and Priors of the method\n\nWe are not sure if we understand what the prior is as in “The method is more like introducing a strong prior specifically for certain data”. Could the reviewer kindly elaborate more on what the “prior” is? If the reviewer means that the method favors structured image data, then we want to emphasize that ReFocus was designed for solving this kind of problems with simple editing processes. Extending to other types of image data is for sure interesting and worth exploring, but a bit beyond our paper’s scope.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The evaluation datasets seem small, why only a subset is selected for evaluation and what is the standard to select data.", "theoretical_claims": "There's no theoretical claim.", "experimental_designs_or_analyses": "1. For the analysis of Refocus to improve grounding and OCR capabilities, it is better to add more experimental results instead of giving conclusions through only two examples.\n2. Lack of analysis for some results, like Table 2.", "relation_to_broader_scientific_literature": "This method proposed a visual CoT approach which uses Python codes as intermediary.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. The motivation of this paper is very clear.\n2. The collected 14k training set is effective.\nWeaknesses\n1. The method is limited and can only be applied to the chart that has a specific structure.\n2. The method is more like introducing a strong prior specifically for certain data, thereby enhancing performance.\n3. The method exhibits significant performance fluctuations across different models, and even different versions of the same GPT model can show substantial variations. Additionally, the performance may have adverse effects on some models, and the author lacks discussion on these points. Meanwhile, please also see the 'Experimental Designs Or Analyses'.", "other_comments_or_suggestions": "No", "questions_for_authors": "Why was the current evaluation dataset selected? For the bar chart, the other datasets like DVQA have more scales of test data and can provide a more effective evaluation. The current testing scale is too small, and the test results also show significant performance fluctuations between different models and datasets. This may stem from the unreliable evaluations caused by the small scale. Meanwhile, due to the strong limitations and priors of the method, the limited data scale can not evaluate the extensibility of the method well.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYMA2J9gwz", "reviewer": "Reviewer_r3TB", "summary": "This paper proposes ReFocus, a visual reasoning framework that enhances structured image understanding with editing on the input image. Specifically, ReFocus iteratively highlights some columns and rows in the tabular problems, masks the other information, and draws bounding boxes on the information that needs attention on. Experimental results show that the proposed method ReFocus yields consistent performance gain across various tasks.\n\n## Update after rebuttal\nThe rebuttal addresses most of my concerns about calling the model \"chain-of-thought\", inference time comparison, etc. One remaining issue that I have is the potential insights in the proposed method that I find hard to evaluate on. The proposed method still seems to be some prompt engineering plus some simple tricks, for me as a person mainly working on general computer vision. Editing something by highlighting [1] to improve performance is also not a brand new technique in computer vision. However, due to my limited knowledge on structured image understanding, I want to request for AC and other reviewers' expertise on this field to tell me whether the proposed method is insightful enough to be accepted to ICML, or it is just some simple tricks and prompt engineering. Therefore, I want to flag this potential issue for AC to confirm the novelty and insight in the proposed method before this paper gets accepted.\n\n[1] Shtedritski et al. What does CLIP know about a red circle? Visual prompt engineering for VLMs. ICCV 2023.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. Below, we provide a detailed response to address the concerns and questions raised:\n\n* Q1. Confusion about “Chain-of-Thought” and “Iterations”\n\nWe would like to clarify any confusion regarding this concept. **ReFocus does not iteratively highlight every column or row.** Instead, it leverages a multimodal LLM to determine which columns or rows to highlight or maskout by generating Python code. For instance, in Figure 1, when provided with an image, the multimodal LLM identifies focus areas (Column “Country”, Column “Wins”) and generates Python code `focus_on_columns_with_maskout(“Country”, “Wins”)` (where  `focus_on_columns_with_maskout` is one of the edit functions, and all edit function names are provided to the LLM in prompt as in Supplementary C. the LLM decides which function to use). The generated edit function is then executed through our backend support. Next, the LLM assesses whether additional edits are necessary (because as in Fig1 it needs two edits), and if so it decides autonomously on the next focus areas. (And this is where the word iteration comes from.) This method allows models to perform self-reason, justifying the use of the term \"chain-of-thought.\" We add \"visual\" to CoT to reflect the presence of intermediate visual editing outputs.\n\n* Q2. Simplicity of the Method\n\nThe difficulty of this method during the prompting stage is specifically how to enable the method to self-reason about what to focus on, rather than relying on iterations. We are the first paper to demonstrate that using simple tools and editing can achieve this goal on structured images. \n\n**Another challenge is determining whether ReFocus can provide a more human-like heuristic supervision signal and whether this supervision is useful in practice.** In our fine-tuning experiments, we found that the training data obtained through ReFocus is 8% more effective than default VQA training pairs and 2.6% better than Chain-of-Thought training data. Overall, both the prompting and fine-tuning goals present significant challenges.\n\n* Q3. Inference Time\n\nNaturally, the inference time increases when incorporating ReFocus compared to using the original base models. It can be seen as roughly twice the inference time, transitioning from Question -> Answer, to Question -> Edit Decisions -> Answer. We did check the model usage, and the generated token number with ReFocus is about twice as that of baseline model.\n\nAnother consideration is cost. For example, on the Chart Horizontal Bar set, the total token count is approximately ten times higher with ReFocus than the baseline setting. This can be too costly for API-based models like GPT-4o. In this context, our fine-tuned model offers a practical alternative. While it may not be as effective as ReFocus combined with GPT-4o, it is cost-free.\n\nWe hope our answers can address the reviewer’s concerns! We’re happy to provide further clarification if needed.", "claims_and_evidence": "Yes, the claim that the proposed ReFocus can help structured image understanding (more specifically, tabular problems) is well validated by the experimental results across multiple benchmarks across tabular understanding tasks.", "methods_and_evaluation_criteria": "Yes, the proposed method makes sense, and I think it is intuitive as it follows the way we humans process the tabular data, as we first focus on several rows / columns that we are interested in, while ignoring other irrelevant rows / columns, then reasoning across the relevant information in the tabular data.", "theoretical_claims": "The paper has no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and valid to me, and also comprehensive as it is conducted across multiple benchmarks and evaluated on state-of-the-art multi-modal LLMs like many versions of GPT-4o.", "relation_to_broader_scientific_literature": "The capability of multi-modal LLMs to understand structured images like charts and figures is broadly related to the advancement towards artificial general intelligence.", "essential_references_not_discussed": "I do not find essential related works that are not discussed. However, this may not be accurate because I am not very familiar with the field of structured image understanding.", "other_strengths_and_weaknesses": "**Strengths:**\n- The proposed method ReFocus conforms to the reasoning process of humans, which looks intuitive to me. \n- The experiments seem to be comprehensive, covering a broad range of benchmarks and structured image understanding tasks and shows the superior performance after incorporating ReFocus on the existing powerful multi-modal LLMs like GPT-4o.\n\n**Weaknesses:**\n- I think it is a bit overclaiming to call the proposed method \"chain-of-thought\". Essentially, the proposed method is iterate the columns / rows to highlight until the multi-modal LLMs determines TERMINATE. Basically, there is no specific \"chain-of-thought\" from the multi-modal LLM agent to find the next most possible rows / columns in a heuristic way by itself with reasoning. Therefore, I think the \"reasoning process\" within this paper is more primitive than the one in the original paper [1], which may be overclaiming to call it \"chain-of-thought\".\n\n- **(I am doubtful of whether it should be a strength / weakness, but I am leaning towards weakness right now)** I often in favor of a method that is \"simple but effective\". For the method proposed in this paper, I think it is effective as supported by the experimental results. However, I think it seems to be too simple to me (because it is basically just iterating the rows and columns to highlight some of them until reaching TERMINATE to get the final answer), which seems more like a simple trick to me, instead of providing insights in the scientific / techincal parts.\n\n[1] Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.", "other_comments_or_suggestions": "I think Section C in the supplementary material is important for understanding how the proposed method works. When I only read Section 3 in the main paper, actually I cannot understand how the proposed method actually works. The information of adopting the \"iterative chain-of-thoughts (although in the Weaknesses part I said I think calling it \"chain-of-thoughts\" may be overclaiming) until reaching TERMINATE\" strategy for conducting the method is important for understanding how the algorithm works. Instead, the main paper spend much space on explaining the different structured image understanding tasks, which I think is unnecessary and can be moved to the supplementary.", "questions_for_authors": "- The main reason for me to give a weak reject as the initial score is that I think the proposed method is so simple that it seems to be a trick for performance improvement instead of providing scientific / techincal insights. What do the authors think of my concerns and arguments? What additional insights the authors want to convey to me about the proposed method?\n- Since the proposed method needs to iterate through the rows and columns to highlight some of them. How is the inference time after adding ReFocus to the multi-modal LLMs compared to the original base models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cDmNTNpZPv", "reviewer": "Reviewer_TSQ2", "summary": "This paper introduces a visual editing method where authors proposed a range of tools (i.e., python functions) that vision language models can use to edit the image (e.g., highlight, draw box, mask out) to better focus on the structured visual content corresponding to the text query. Experiments show that this process improves model performance on certain tabular and chart understanding tasks, and authors show that open-weight vision language models can also benefit from this process by being finetuned on data that involves tool calling that leads to the correct answer.\n\n## update after rebuttal\nI would like to thank the authors for their rebuttal and thoughtful follow-up addressing my concerns and questions. I have carefully re-evaluated the manuscript, particularly Section 5, and respectfully disagree with the authors’ claim that their method demonstrates the ability of models to generate novel functions for visual editing at test time (which was the core of my initial question). In my view, providing function definitions as part of the prompt, or generating answers in the same structured format as used during training (e.g., `<image> <question> <thought1> <ReFocus bounding box> <thought2> <answer>`), does not constitute novel test-time function generation. Rather, it reflects learned behavior from training patterns, not evidence of the model inventing new visual editing logic on its own. Because authors did not show that models can generate novel visual editing functions, the answer to the follow-up question \"can models make good use of these self-generated functions to edit the image and improve their performance\" remains unclear. \n\nThat said, I do agree with the authors that the proposed prompting approach is effective, and that fine-tuning models with rejection-sampled visual CoT data distilled from trajectories does lead to improved performance. However, I still remain hesitant in considering the work to be above the bar for a conference paper due to the limited scope and a relative lack of novel technical contributions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback. Below, we provide a detailed response to address the concerns and questions raised:\n\n* Q1. External Expert Knowledge Comparison with Visual SketchPad\n\nWe would like to clarify some confusions here. What we want to say is that, the tools used in ReFocus, do not bring in external information – every information is constrained to what’s provided in the input images. However, in Visual SketchPad, specialists such as Depth Anything will bring in external information (e.g. a depth map)  to the process, to assist in decision making. We'll ensure this distinction is clearer in our revisions.\n\n* Q2. Claim about Lack of multi-hop visual reasoning ability\n\nBasically, we were referring to all different kinds of multimodal LLMs (e.g. GPT, Gemini, Claude, LLaVa, LLaVa-Next, etc.), when we ask a VQA question, they could not answer while providing the selective attention areas or multi-hop visual reasoning processes. When we play with these models, they would at most generate some thinking processes in text, but not in visual format, on structured images, as we explained in the introduction. Many related literature are listed in Related Work starting line 154. But we were just making a very general claim through our observations, that multimodal LLMs cannot return the visual area to focus on during question answering. \n\n* Q3. Analysis Constraints\n\nWe will update in our revisions to constrain our conclusions toward a particular dataset for a particular model when discussing a particular editing method that works the best. \n\n\n* Q4. Contribution & Comparison with Visual SketchPad\n\nWe would like to clarify any confusion regarding this concern. ReFocus is the first method to excel at structured image tasks, which Visual SketchPad cannot solve. **More importantly, ReFocus explores whether it can provide a more human-like heuristic supervision signal and if this supervision is practically useful—unlike Visual SketchPad, which is limited to prompting only.** In our fine-tuning experiments (Section 5), we found that training data obtained through ReFocus is 8% more effective than default VQA training pairs and 2.6% better than Chain-of-Thought data. We believe our fine-tuned model aligns with the reviewer’s definition of “performing on-the-fly refocus without relying solely on a few tools”.\n\n\n* Q5. Cost / Speed Trade-off Analysis\n\nDue to lack of time, we conducted a model usage analysis on Chart Horizontal Bar Set. \nAs for the consideration of speed, the generated token count with ReFocus is about twice as that of the baseline model.\nAs for the consideration of cost, the total token count is approximately ten times higher with ReFocus than the baseline. This can be too costly for API-based models like GPT-4o. In this context, our fine-tuned model offers a practical alternative. While it may not be as effective as ReFocus combined with GPT-4o, it is cost-free.\nWe will include these discussions in our revisions.", "claims_and_evidence": "- I think that the claims are well-supported in general, though the comparison with Visual Sketchpad that it incorporated external expert knowledge while ReFocus does not require additional information does not appear to be fair. In particular, ReFocus still relies on algorithmic tools (i.e., `findContours` , `getStructuringElement` ) to extract coordinates of table/chart element, which weakens the argument that it doesn’t rely on additional information. To really support this claim, an ideal setting would be that the model needs to provide the coordinates as input arguments (instead of providing any labels to mask and let the algorithm decide the region).\n- For your claim that multimodal models lack selective attention and multi-hop visual reasoning ability mentioned in the introduction, can authors provide supporting literature and/or additional experiments?", "methods_and_evaluation_criteria": "Yes, the selection of evaluation dataset appears to be suitable within its application scope.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "On the analysis that discussed the particular editing method that works the best, authors should constrain their conclusion toward that particular dataset for a particular model. The conclusion might not be generalizable on other datasets or other models.", "relation_to_broader_scientific_literature": "The general finding aligns with Visual Sketchpad where visual editing improves vision language model performance.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "- The paper is clearly written, which I really appreciate.\n- I think the contribution is too incremental and limited in scope. In particular, I believe ReFocus is simply an analogy to Visual Sketchpad used for structured images (with a different set of tools). Because all the functions are pre-defined and the functionality is hardcoded, the experiments can only be done with very few datasets that show compatibility with these handcrafted tools. It’s not a general method that improves vision language model on a wide range of datasets, not even to say applications. I would be much more convinced if you can show that models can generate code on-the-fly to achieve some “focus” functionality and apply it to a variety of structured image evaluations instead of sticking to a few hand-crafted tools on very few evaluation settings.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "- I’m curious if you can provide any cost / speed analysis in inference because apparently I think there’s a trade-off between better accuracy and prolonged generation in thoughts and IO.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents ReFocus, a method that lets vision-language models edit images (like highlighting or masking) to improve structured image understanding tasks such as tables and charts. The idea is to mimic visual chain-of-thought by letting the model focus step by step. Reviewers generally liked the paper, and AC also likes the idea. It is clearly written and shows good performance gains. One strength is that ReFocus not only helps prompting but also provides better training data, improving model fine-tuning. Reviewers' concerns span from small evaluation datasets and limited generality, but authors addressed these with a diligent attempt, showing comparisons to related works and discussing future extensions. AC also feel calling it “chain-of-thought” might be too strong, but after author clarification, I feel it is also okay... AC feels having a logo in front of the title is disturbing, and may cause paper parsing engines to introduce errors for archiving, please consider removing it. 

Overall, this is a useful and well-presented paper with practical impact of interest to ICML community. I recommend a weak accept.
Decision: Accept (poster)