Chain-of-region: Visual Language Models Need  Details for Diagram Analysis
Authors: Xue Li
Yiyou Sun
Wei Cheng
Yinglun Zhu
Haifeng Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-modality; Visual Language Model; Computer Vision
Primary area: foundation or frontier models, including LLMs

Abstract: Visual Language Models (VLMs) like GPT-4V have broadened the scope of LLM applications, yet they face significant challenges in accurately processing visual details, particularly in scientific diagrams. 
This paper explores the necessity of meticulous visual detail collection and region decomposition for enhancing the performance of VLMs in scientific diagram analysis. We propose a novel approach that combines traditional computer vision techniques with VLMs to systematically decompose diagrams into discernible visual elements and aggregate essential metadata. Our method employs techniques in OpenCV library to identify and label regions, followed by a refinement process using shape detection and region merging algorithms, which are particularly suited to the structured nature of scientific diagrams. This strategy not only improves the granularity and accuracy of visual information processing but also extends the capabilities of VLMs beyond their current limitations. We validate our approach through a series of experiments that demonstrate enhanced performance in diagram analysis tasks, setting a new standard for integrating visual and language processing in a multimodal context.

Published as a conference paper at ICLR 2025
CHAIN-OF-REGION VISUAL LANGUAGE MODELS NEED DETAILS FOR DIAGRAM ANALYSIS
Xue : Yiyou Sun Wei Cheng', Yinglun Zhu', Haifeng Chen
Universityy of Wisconsin-Madison, UUnieersity of California, Berkeley, NEC Laboratories America Universiit of California, Riverside
ABSTRACT
Visual Language Models (VLMs) like GPT-4V have broadened the scope of LLM applications, yet they face significant challenges in accurately processing visual details, particularly in scientific diagrams. This paper explores the necessity of meticulous visual detail collection and region decomposition for enhancing the performance of VLMs in scientific diagram analysis. We propose a novel approach that combines traditional computer vision techniques with VLMs systematically decompose diagrams into discernible visual elements and aggregate essential metadata. Our method employs techniques OpenCV library to identify and label regions, followed by refinement process using shape detection and region merging algorithms, which are particularly suited to the structured nature scientific diagrams. This strategy not only improves the granularity and accuracy of visual information processing but also extends the capabilities of VLMs beyond their current limitations. We validate approach through series of experiments that demonstrate enhanced performance diagram analysis tasks, setting a new standard for integrating visual and language processing in a multimodal context.
INTRODUCTION
Recent advances in Visual Language Models (VLM), such as GPT-4V (OpenAl 2023) and Gemini
(Team et al., 2023), have demonstrated broad applicability in multimodal downstream tasks, including navigation Yan et al. 2023), search (Cloud (2024), and answering scientific questions (Yue et al.
2024b). Despite these achievements, significant limitations persist in the perceptual capabilities of
VLMs. They often fail to discern fine visual details (Togg et al., 2024 Peng et al.) (002)), are easily misled by visual distractors (Zhang al., 2024b), struggle with understanding visual relationships (Yue et al. (2024b). Fig. 43), and may even hallucinate non-existent objects (Huang et al. (0244). notable example is shown in Figure (a),
(a), where GPT-4 cannot accurately determine the specific value from a bar graph. Such perceptual flaws substantially impede the ability of LLMs to perform
rigorous scientific analysis on diagrams.
Addressing these perceptual issues raises questions about the capacity limit of VLMs. It remains uncertain how much visual instruction data is necessary (Zhang et al., 2024a Liu et al. 2024b) or whether it is feasible to develop VLM that can impeccably manage visual details in complex diagrams given computational constraints (Bai et al. 2024). A promising approach to overcoming these limitations involves sequential inference of visual elements, rather than generating responses in a single pass. This iterative strategy is previously validated in works like Tree-of-thought (Yao) et al. 2024). Unlike tasks expressed in natural language. visual tasks often necessitate further decomposition into specific regions. This paper focuses on the following critical research question:
How can regions be decomposed and visual details collated for scientific diagram analysis?
One straightforward approach involves using segmentation methods such as SAM (Kiilllvv et al. 2023) to generate regions of interest with a segmentation mask. This strategy, used in recent studies (Yang et al. 2023 Yan et al. 2024), tags each object with a number along with the segmentation mask before VLM analysis. However, when applied to scientific diagrams, the accuracy of segmentation
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Summarize the bar valuee.
Summarize the bar values.
the yrici raigut fre IJ cel bnfonee p0-k)
Region [Ac Two vertical bers:
Ractangular: (kmin 1018, ymin 134,
1107, ymexx 754)
Rectangular: (xmin 1120, ymin
1215, ymax
cacclatt the piiiiit lal ritis yrass
Region [B]: a-axss with numbers from 82-100
Jectiac conerr ploa posittic to potl
Masch Hirlay
Bodkay
Mash
Minearhh
sitel, 19 peita
aNoke DEgert
Region [P]: Background Lines with Y- coordinates from 59 7751
Match Hockey:
.....
Novice: Expert
Chain-of- Region Prompt
To find the average value, we to consider the visual values indicated on the chart:
Mismatch Hockey:
Koday Bostet
Nevica: 86 Exprrt: 82
.....
Nonhockey:
proceed, I'll analyne pixel hoights of the bars relative y-axis values provided in the image.
Mismatch Nonhockey:
Novice: 95 Expert B4
Mismatolh Hockey
Mittch Hockey
Aoodioe Expert
Mittch Noshockey:
Milematch Nonhockey:
Nouioe Sl Expert
-Nviow Expert
Novioe: 58 Expart
Novioe C4 Expert
(a) Original VLM fails capture visual details
(b) Augmented VLM with Chain-of-region (CoR) Prompt
Figure 1: Comparison of GPT-4o (OpenAl 2023) and the one augmented with a Chain-of-Region (CoR) prompt in addressing queries about scientific diagrams. (a) illustrates the difficulty GPT--4o encounters when directly inferring exact values from bar graph. (b) demonstrates how additional visual details (such pixel location background lines and rectangular parameters) along with the original figure can be further utilized to accurately infer the accurate bar values.
methods is limited by several factors: a) generalization issues, as methods trained on natural images often fail with -itttr-ddittibutioon input such scientific diagrams (details in the Appendix [B): b) difficulty in detecting small or thin objects, such as lines or dots in charts; and c) the inability to infer hidden shape parameters, such as the rotation angle of rectangle or the slope of line.
To overcome these challenges, we propose a novel integration of nearly forgotten yet potent traditional computer vision techniques (commonly implemented in OpenCV (Bradski) (0000) with VLMs to initialize (Section 3.1.I). split (Section (.1cctoon....)) 3.1.2) and merge (Section 3.1.3) regions, and collect meta- information (Section 3.2). The effectiveness of traditional techniques hinges on a salient observation of scientific diagrams: unlike natural images, these diagrams have homogeneous colors and structured patterns for each visual element, as they are human cerman-created by software or programs. For example, we flexibly utilize OpenCV operators, such as cv2. connect edComponett for initial region separation and cv2. f ndContours to generate contours for shape detection.
We propose the Chain-of -fe---oooononnn (CoR) framework for diagram analysis, which first integrates traditional CV techniques to decompose the diagram to collect visual details and then incorporates these visual details into VLM for a fine-grained analysis (e.g.. Figure [b)). Our CoR framework has the following compelling advantages:
1. White-box Algorithm: Unlike deep learning models that operate as black boxes, the CoR framework employs transparent, rule-based region separation methods. This white-box approach allows for greater interpretability and easier debugging, enabling users to understand and modify the processing steps based on specific requirements of the task.
Compared to DL-based methods, the CoR framework is simpler and more cost- effective. It requires only CPU processing, and its operations, which are less computationally intensive, can be executed in milliseconds.
Cost-Effective:
3. Plug-and-Play Compatibility: The CoR framework is designed for seamless integration, which can be effortlessly incorporated with any pre-trained VLM. This plug-and-play compatibility ensures that users can leverage existing VLMs without extensive retraining or adaptation.
4. Strong Empirical Performance: In the realm of scientific diagram analysis, the CoR framework demonstrates superior performance. Extensive evaluation benchmarks show that it not only enhances the accuracy of visual detail extraction but also improves the overall reliability of the analyses conducted by VLMs. These benchmarks indicate that CoR significantly outperforms existing DL-based methods in terms of precision and recall in complex diagram interpretations.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Augmented Prompting for VLMs. VLMs (Dosovitskiy et al. 2021 Zhu et al. 2023
OpenA 2023
Liu et al.) 2024b a Li et al. 2024a) enhance LLMs by integrating visual perception capabilities. To improve their handling of complex images, various prompting techniques have been developed.
Iterative strategies, such as those seen Tree-of-thoughts (Yao et al. 2024), employ visual tags (Yang
et al. [023 Nasiriany et al., 2024). Hierarchical Layout Tree(Fan et al. 2024b), Language Model- guided Diagram Parsing Tools (Wang et al. (024) to guide VLMs in decision-making processes. Additionally, some researchers employ external tools enhance VL.M prompts. Yang et al. (023) introduces Set-of-Mark (SoM) Prompting, utilizing segmentation masks to tag regions, while Wu et al. 2024) develops a toolkit that focuses VLM attention specific regional details with aids like rulers and compasses. Lu et al. (2024) streamlines the integration of image caption tools and text detectors into visual prompting, highlighting the ongoing innovation in VLM applications. Our method integrates traditional CV techniques with VLMs to enhance prompting capabilities further.
Enhancing Spatial Understanding in VLMs. Recent studies in enhancing VLMs explore various
modifications to improve spatial reasoning and detail recognition. Some research focuses on altering the existing architecture incorporate spatial input like bounding box (Liu et al. 2024c). learnable spatial CoopppDooreewaar
et al. (2024), segmentation masks (Guo et al. 2024). 3D voxels (Man et al. 2024. Chen et al. 2024), depth information (Cheng et al. 2024) and RGBA region-text pairs (Sun) et al. 2024). On the other hand, several studies aim to augment VLMs by using additional tuning data Yan et al. (2024); Liu et al.] (2024b ay; Xuan et al. (024);; Sun et al. (2024); Chen et al.
(0024) ;
Li et al. (2024b). This approach is evident in the work by Zhang et al. (2024c); Meng et al. (2024)
Han et al. (2023); Fan et al. (2024a) who gather visual instruction data specifically on charts and fine-tune VLMs for better performance in interpreting such data. While these approaches necessitate fine-tuning procedures, heavy human effort dataset preparation, and often face generalization challenges, our method integrates traditional CV techniques with VLMs more efficiently handle complex scientific diagrams.
METHODOLOOGY
This section outlines the Chain-of- ofii--iooRR-ooooeee pipeline, as it reflects the sequential processing of separate visual regions to form cohesive analysis chain, and its application in enhancing the perceptual abilities of VLMs for handling complex scientific diagrams.
Overview. Typically, a VLM denoted as F, takes an image Xim E #xxx3 and a text query X₁, and generates a textual output Ý ~ F(Xim, Xx) aimed to closely match the desired response Y. However, it is counterintuitive, even for humans, to answer complex questions, such as the one in Figure D with just a single glance (or one function pass). This often requires detailed visual assessment of each component represented in a diagram, such as reading bar values itcratively.
To mimic this methodical process, employ a divide-anddconquee approach that iteratively extracts and compiles detailed region- based information into the VLM's prompt. As demonstrated Figure 2 this process consists of two primary steps: (a) Region Identification (Section This step involves designing an algorithm to identify distinct regions within the image. These regions correspond to discrete visual elements or clusters of elements relevant to the text query. (b) Information Assembly (Section 3.2): Once the regions are identified, this step focuses on extracting specific information from each region. The extracted data are then systematically aggregated and integrated into the VLM prompt, enhancing the model's ability to generate accurate and contextually relevant responses.
3.1 REGION IDENTIFICATION
At high level, the Region Identification process includes three stages: initialization, splitting, and merging, as illustrated in Figure 2 The pipeline begins with Region Initialization using the method to identify initial regions (Section 3.1.I). demonstrated by separating a graph into 16 distinct regions in Figure 2 a). However, certain visual elements, such as lines and circles, may remain unseparated due to being connected components, posing challenges in isolating individual parameters. These regions then undergo Region Splitting (Section 3.1.2). where both structured and unstructured splits performed. During this stage, critical parameters such as
3

---

Published as a conference paper at ICLR 2025
Section 3.1: Region Identification
Image Input
3.1.1 Region Init.
3.1.2 Region Split
3.1.3 Region Merge
cv2. w22.commecteddoorpoeentt
Structured Split
Panama Y.Y.W.
Structed Merge
Backgrmand meee
P
P
P
P
P
S
Unstructured Split
Unstructured Merge
Lines
Teck
Section 3.2 Information Assembly
syplcoerat livd www.eaaaattiaattt..ooo (ittttte
Structure Info
2
Crrde 1552, 7931, Redia 130
3
Anow: (461, 521) la (651. 720) Anow: (405, 763) to (183, 515)
Semantic Info
node labiled 'P_ with one pointing
(a) Initial Region
Regions
Regions
Figure 2: Overview of the Chain-of-Region (CoR) framework. Fach cray-houndary box represents an individual region. In (a), the initial regions are identified. Moving from (a) to (h), one large region is split into circles and arrows, using structured splitting, From (b) to (c), all existing regions merged into distinct clusters. In red box, we exemplify information assembly process with a merged region containing P4 node.
pixel locations, width, height, and angles extracted for further analysis. As strategy focuses on sequentially accessing regions to gather detailed information, we must consider the budget limit on the number of regions that can be sent the VLM for inference. To address this, Region Merging (Section 3.1.3), regions are merged by considering various structural elements like background lines and captions. These regions are further clustered by egglomeratiiecCoo the number of regions follows the budget limit.
ng to ensure that
3.1.1 REGION INITIALIZATION
At the first stage, initialize the region using connected components, based the observation that visual elements in scientific diagrams exhibit homogeneous colors and structured patterns. This process begins with the conversion of the input Xmm into a binary image Xbi, facilitating the separation of distinct visual components. The code snippet for this process is as follows:
X_ cv2. chreshold(X_im, 0, 1, cv2. THRESH_OTSS OTTU)
X_fg cv2. wwwneeteddoooooeeeeeeoooooeeeeee (X_bi)
__ cv2. wet.ccnntedooooooeeettttt (1 - bi)
regiion = f_ x x_bg-
(X_bg > 0) * offset
First, the image is converted into grayscale and then into a binary format using a thresholding technique. The cv2. threshold function applies Otsu's thresholding method (Otsu al.) 1975), which automatically determines the optimal threshold value for separating the image into foreground and background. The resulting binary image Xu. contains clear distinctions between the visual elements and the background.
Next, we utilize the cv2. ConnecttedComponens: function to identify and label the connected regions within Xy.. The connected components algorithm assigns a unique label to each connected region based on 8-connectivity[] The output, Xfg- represents the initial segmentation of the diagram where each unique label in the pixel location corresponds to a different visual component.
Additionally, we process the inverse of the binary image, Xbi, to capture any components that might have been categorized as background in the original binary segmentation. This step ensures that regions which are visually connected but appear as dark on a light background are also identified as distinct entities. The result, Xbg. provides complementary set of region masks.
The final step involves combining these X/9 and Xbg to create unified region map, Xreg- The term (Xag 0) offset ensures that the labels from Xbg adjusted by specified offset that guarantees
'8-oonectivvtyy refers the concept in image processing where pixel is considered connected to its eight surrounding pixels, including diagonally adjacent ones.
4

---

Published as a conference paper at ICLR 2025
that there are overlapping region labels, since both region maps begin the labeling from zero. Together, these steps form strategy for segmenting scientific diagrams into distinct and analyzable regions, setting the stage for further processing steps involving region splitting and merging.
3.1.2 REGION SPLIT
The initial region identification described provides basic segmentation of visual elements. However, as illustrated in Figure this method may not adequately separate connected objects with overlapping or connected elements. To address this limitation, we introduce two approaches: structured split and unstructured split.
General Split Procedure. We first introduce the general region-splitiing procedure. The following Python function outlines the core of our splitting process:
def M. remain = cv2.subtracc (M, M sub)
ons (M, M_sub):
_, remain ev2. comeetedCompoeentss (M remain)
region update region (X regicn, X remain)
This function takes two masks: M, the original mask the region, and Msub. a mask representing subset of M that needs to be separated. Using OpenCV's subtract method, we remove Mau from M to isolate the remaining parts of the region. We then apply connectedComponentt identify newly isolated components within the remaining mask, Mremain- These components are integrated into the existing region map region through the update eergeiinnn function, which effectively updates our overall region separation by incorporating the new distinct regions.
VLM-Assisted Structure Recognition. Before proceeding with specific region splitting, ploy a VLM to preliminarily discern the potential structures or shapes within each main region, prioritizing regions based their sizes. This discernment step, detailed further in the Ap- pendix A.1 involves a prompt-based query to the VLM which suggests possible structural categories. The VLM's response then directly triggers the corresponding structure shape detector. If the VLM does not recognize the structure as one present in our current library, the process defaults to an unstructured component split. Note that this discerning step is constrained several main regions with large area size and limited by a pre-defined recognition call limits (ex. 10) per diagram.
Structured Split. For regions identified as containing pre-defined structures, the splitting process specialized to handle standard forms, such as rectangles, ellipses, and lines. We employ structure detector that processes each region independently based on its labeled mask (M₃ E W
shape, S = structure dew.ueoreeetcoooooeeoe (M split regions (M i, M. shape)
i: the shape parameters
The function st ructure detector analyzes M, and identifies sub-regions M Msaape that corre- spond to distinct structured elements within the region, along with their parameters S, The detected sub-regions are then split from the main region using the described split_reegiee
function.
We include a set of common structures typically encountered in the analysis of scientific diagrams (detailed in Appendix A.2). This includes both third-party algorithms such the ellipse detector (Meng et al. (220)) and the segment detector (Von Gioi et al. 2008), as well as our own implementa- tions for detecting rectangular shapes. These structure detectors enable the extraction of accurate meta-informattio,, such as the pixel location of rectangles, which are critical for detailed diagram analysis.
Unstructured split. Dealing with unstructured shapes presents a distinct challenge compared to structured shapes,
impractical to design a "universal" computer vision algorithm that accom- modates all visual elements effectively. Our unstructured split algorithm stems straightforward intuition: composite visual elements can often be deconstructed into their constituent parts based on "connectivity" For instance, consider the example depicted in Figure (a), where cup shape shows a weakly connected spot at the juncture between the bottle and the stick. This intuition underpins our methodology.
It be obtained comparing the region map X Xregion against the label
X Xregion
5

---

Published as a conference paper at ICLR 2025
Unstrrch ct uned
Shagee
(Low tenewon))
Witer
Fill Glue (High tonsion)
Raw Image (Part)
Region Infia tcation
Unstractured Split
(8) Unstructured Split: An Analogy Filling Liquid
(b) Demonstration Unstructured Split
Figure 3: Illustration of the unstructured split principle. (a) The splitting process is analogous to filling high-tension liquid into container, where the shape is divided into filled (yellow) and unfilled (white) regions. (b) real-work example demonstrating the separation an unstructured shape "handle" from hand. Different colors represent different separated regions.
Our algorithm can be viewed as the process of filling a high-tenssee high-tension liquid within the composite shape, thereby identifying natural divisions between the filled and unfilled regions, simple implementation of this concept can simulate this filling and splitting process:
2 # T: tension force kernel cv2 getttruttuuigggl ement: (cv2. MORPH ELLIPSE, (T, T))
= cv2. www.iiittneeeaaassssaoooo (M_i, cv2.DIST __]___ max () //22
M_shape cv2 cv2.morre morphologyEx (M_i, cv2. MOORHOORRPEPN,, kernel, iterations=.)
(M_i, i, M. shape)
This code snippet outlines our algorithm's operation. The cv2 . wwt..diaaaneerrasffrrr..e function calculates the distance the nearest zero pixel for each pixel of the binary image M, (the mask of the current region). The maximum distance found is halved to define a tension threshold T, which serves basis for the ellipse-shaped kernel. This kernel is used in cv2 mww.hollrooooooo logyEx with cv2 MMRPMORPHERE PORPP_OPPA EN, which separates the image based on the defined tension areas. The morphological opening helps to isolate regions within the image by breaking down weak connections, as simulated by our high-tension liquid analogy. This unstructured split technique is advantageous for analyzing scientific diagrams where shapes may not conform to standard geometric norms but still exhibit distinct connectivity properties that can be exploited for detailed analysis.
3.1.3 REGION MERGE
After splitting, we often encounter scenarios where hundreds of separated regions have been identified. Given budget constraints, it is impractical to extract information by querying the VLMs for each region individually. Therefore, in this section, we discuss methods to reduce the number of regions by merging several smaller regions into more comprehensive ones.
Structured Merge. The first approach to region merging is structured, relying on both existing tools and heuristic rules. (a) We utilize mature OCR tools (Contributors, 2024) detect and generate caption boxes within the diagram. Regions that are enclosed within these text boxes are considered related and are merged into a single region; (b) We apply a set of heuristic rules merge regions based on visual patterns that follow fixed rules, such as dotted lines and background lines (details in Appendix A.3).
Unstructured Merge. After the structured merging, we also employ an unstructured approach further consolidate regions, especially when dealing with complex diagrams where structured methods may not suffice. Specifically, we apply hierarchical clustering (Nielsen & Nielsen 2016) to the region masks. The clustering process is governed by distance matrix D) where each element Dij represents the distance calculated based on the centroids of the region masks M; and My. The hierarchical clustering process will stop when the number of merged regions reaches a predetined budget B. This budget acts as threshold to balance the number of queries made to the VLM against the need to maintain a comprehensive understanding of the diagram. As a result, the final output set of region masks will be {M, I where each M represents a merged region ready for further analysis with the VLM.
3.2 INFORMATION COLLECTION
The final phase focuses the collection of meta-information from each region M

---

Published as a conference paper at ICLR 2025
Structure Information Collection. First, collect shape information S; for each region M2, which has been previously collected in the structured split phase. This information includes details about the geometric properties of the region such as contours, area, and perimeter.
Semantic Information Extraction. Next, we extract semantic information that includes the role and entity type of each region, along with detailed description. To facilitate this, we visually highlight the region in image X masked with M and fading other regions. It ensures that the VLM's attention is directed towards the relevant region during the analysis. The process for this visualization and the subsequent query construction is detailed in the Appendix A.4] The highlighted image is then sent the VLM using a specific prompt designed to extract detailed semantic information.
Aggregation and Query Formulation. The collected shape and semantic information for each region aggregated and appended master query Xg. This query, enriched with detailed regional metadata, is subsequently used to generate a response Y from the VLM. The process is illustrated in Figure g in Appendix, which shows how each piece of collected information contributes to the formulation of the query and influences the VI.M's output.
By systematically collecting and integrating both shape and semantic information, let VLM not only recognize the physical attributes of the diagram components but also understand their contextual and functional utility.
4 EXPERIMENTS
Datasets. We conducted extensive experiments on the Massive Aulti-discipline Multimodal (MMMU, Yue et al. (2024)) dataset, which includes a diverse array of multimodal questions sourced from college exams, quizzes, and texthooks. These questions combine image inputs with textual inquiries. While our network is specifically tailored for scientific diagrams, MMMU encompasses a broad range of natural images across categories such as Art & Design. To align our research focus, we conduct experiments on several sub-categories of MMMU that consists exclusively of scientific diagrams. Specifically, this subset includes images from two main categories: Science (encompassing Biology, Chemi stry, Geography, Math, and Physic i cs) and Tech and Engineering (Architecter & Engi neeri Computer Sci ence, F. electrnn cs, Energy & Power, Material and Mechani ca Eng i neering). This tailored subset comprises a total of 5,210 images.
Implementation Details. We evaluate the performance of the main-stream pre-trained LLMs using GPT-4, GPT 4o, and GPT mini backbone models. Specifically, we utilize gpt-4-turbo, chatgpt-4o-latest and spt-4o-mini-2024-00 18 respectively. The primary hyperparameters employed in our Mlain-of-Region method include the pre- pre-defined recognition call limits in split step and the cluster number during the unstructured merge step, detailed in Sections Sections|3.1.2 (3122 and 3.1.3] In the current version, let the base model used in the intermediate structure recognition call be the same the one used in the final prediction. We set the recognition call limit to 10 and the cluster number to 5. We also provide the sensitivity analysis in this section. While other hyperparameters exist for structure split/merge steps, we adhere to either the default parameters provided by OpenCV or those commonly used in existing GitHub libraries. Given their minor impact on the final algorithm, we do not engage in an extensive discussion on these parameters.
4.1 MAIN RESULTS
Baselines. We evaluate diverse strategies for prompting VLMs for fine-grained analysis:
1. Raw VLM model: We sent the image and question directly into VLM to generate the final answer.
2. Zero-shot CoT (Kojima et al. 2022 enhances the LLM's inference capabilities by including the phrase "Let's 's think step by step" in the prompt. We adapt this technique to the multimodal QA setting by modifying the prompt to "Let's summarize information region by region before answering the question."
3. Few-shot CoT (Wei et al. 2022 leverages examples within the prompt to further improve LLM reasoning. In our approach, we incorporate examples of region descriptions in Appendix A.5
AAlis for Chain-of-Thought.
7

---

Published as a conference paper at ICLR 2025
4. Meta Segment Anything Model 2 (SAM2) (Research 2024) offers commercial- segmen- tation tool to supply region masks. We employ this technique as baseline for comparing segment results with proposed CoR method. Specifically, we utilize the "everything" mode to provide initial segment pieces, applying the unstructured chustering/merging strategy outlined in Section 3.1.3 to produce the final regions, which are then processed in the information collection phase described in Section
3.2
SOM Yang et al.
2 appends numbered tags to regions segmented by SAM2. The entire tagged image then sent to the VLM for inference.
CoT helps improve better performance in multimodal scientific question answering. In Table [ we present the primary results of our experiments, which compare the proposed Ohain-of-Region (CoR) methods against baseline models (GPT4o-mini, GPT4-Turbo and GPT-4o) across various tasks. Our observations are as follows: a) Zero-shot CoT and Few-shot CoT (Wei al.) 2022) enhance the VLM's inferential capabilities by eliciting a thought process that guides the VLM through problem-solving. b) SoM (Yang et al. 2023) proves to less effective, primarily because it was originally designed for natural images, not scientific diagrams. The tags used in SoM can often mislead the VLM, lcading to confusion and incorrect inferences. c) We also evaluated the baseline method SAM2 (Research) 2024), which employs the same inferential strategy sequentially summarizing regions, albeit using different segmentation masks. The results indicate that SAM2 is highly competitive with other baselines, underscoring the effectiveness of gathering information region-by-region prior to responding. However, CoR outperforms SAM2 by utilizing superior regional masks that better isolate and identify relevant features within diagrams. Overall, the CoR demonstrates its superior ability to handle complex visual information and contribute to more accurate VLM inferences.
by
Hyperparameters and Sensitivity Analysis. In our framework, two key hyperparameters are pivotal: a) the predefined recognition call limit in the split step as detailed in Section and b) the number of clusters during the unstructured merge step, which is elaborated in Section 3.1.3] In the current implementation, we have set the recognition call limit to 10 and the number of clusters to 5. To evaluate sensitivity, we varied the recognition call limit within the range of [3, 5, 10, 15, 20] and the cluster number within the range of [1, 3, 5, 10, 20). We maintained other hyperparameters constant to isolate the effects of each tested variable.
The performance comparisons for each hyperparameter are illustrated in the bar plot in Figure 4 From this analysis, observe that the recognition call limit has a relatively minor influence on the final outcomes. In contrast, the number of clusters significantly impacts performance as it controls the granularity of details which the VLM attends. As shown in the bar plot, optimal performance is achieved at a moderate level of granularity; performance improves initially with an increase in cluster number but deteriorates when the number becomes excessively high. It aligns with the intuition: if the cluster number is too low, the VLM may overlook critical details; conversely, if it is too high, the VLM may become overwhelmed with minutiae and lose sight of the overall structure. Our results suggest that the ideal number of clusters should be tailored to the complexity of the input diagram. Dynamically selecting the cluster number opens an avenue for further research.
Average Accuracy
60
Average Accuracy
60
50
55
51.5
51.2
52.9
50.1
51.5
50
55
46.4
50
52.9
53.2
49.4
45
45
40
40
35
35
3
10
15
20
3
5
10
20
Recognition Call Limit
Cluster Number
Figure 4: Sensitivity analysis of hlppe-parameters. on MMMU dataset. The average accuracy is reported on the
MMMU dataset with backbone GPT-4o. The middle bar corresponds the value used in our main experiments.
4.2 ANALYSIS ON SEGMENTATION RESULTS
In the realm of scientific diagram analysis, the precision in segregating each regional visual element is crucial. This section aims to compare the segmentation results of our method against Segment Anything Model 2 (SAM2) (Research) 2024), the state-of-the-an segmentation model but primarily
8

---

Published as a conference paper at ICLR 2025
Table 1: Main results for QA on scientific diagrams across the MMMU dataset. Bio., Chem.,, Geo.. Arch., CS, Elec., Mater. and ME short aliases for Biology. Chemistry, Geography, Architecture Engineering, Computer Science, Electronics, Energy Power, Materials, and Mechanical Engineering
respectively.
Science Geo.
Tech and Engineering
Method
Bio.
Chem.
Math
Physics
Arch.
CS
Elec.
Energy
Mater.
ME
Average
GPT4o-mini
40.9
35.5
52.3
47.2
40.0
25.8
33,4
34.6
30.6
42.0
31.9
37.6
Zero-shot CoT
38.4
32.6
52.1
38.7
45.7
29.2
35.1
23.1
34.7
42.4
44.4
37.9
Pew-shot CoT
31.3
37.7
50.9
45.1
42.3
30.4
45.5
31.0
47.5
38.7
35.0
39.6
SAM2
38.5
40.0
54.8
52.2
47.9
40.5
34.7
35.3
25.3
45.5
41.9
SoM
31.2
33.6
40.0
43.5
45.1
35.2
27.0
17.9
41.2
34.2
29.9
34.4
CoR (Ours)
42.0
34.6
51.8
51.4
52.8
45.1
43.3
33,5
48.7
44,9
51.9
45.4
GPr4-Turho
41.5
34.8
51.0
41.6
49.9
47.0
57.4
34.6
41.5
45.0
32.5
43.3
Zero-shot CoT
43.3
25.1
59.6
46.5
58.3
44.8
50.8
31.3
46.8
31.4
50.0
44.3
Few-shot CoT
40.0
37.8
51.8
52.8
52.4
51.6
57.9
47.6
45.2
32.2
51.7
47.4
SAM2
38.3
39.9
48.4
63.4
48.0
44.2
50.6
48.6
66.9
37.9
49.3
48.7
SoM
41.0
30.3
44.8
47.6
44.0
42.3
37.5
15.1
42.1
26.9
24.7
36.0
CoR (Ours)
42.9
43.5
56.0
56.8
61.0
51.9
55.7
52.4
65.2
46.0
57.0
53.5
GPT40
41.3
33.6
45,4
51.6
28.8
41.7
24.6
45.1
22.2
37.7
Zero-shot CoT
47.7
46.1
47.4
52.2
66.6
44.3
50.6
29.3
41.5
42.6
47.5
46.9
Few-shot CoTT
47.8
38.6
58.0
44.9
56.1
51.3
35.2
34.2
41.8
36.5
42.3
44.2
SAM2
44.7
42.3
41.5
51.6
64.4
42.0
48.8
38.0
48.9
48.9
45.8
47.0
SoM
48.9
37.3
44.7
49.8
25.9
38.9
28.1
57.5
33.9
24.6
40.4
CoR (Ours)
49.8
46.7
59.1
53.9
72.3
49.7
49.9
34.7
65.7
41.0
59.1
52.9
tailored for natural images, not scientific diagrams. SAM2 often overlooks small or slender objects, which are common scientific diagrams. This section provides comparison both quantitatively and qualitatively, highlighting the effectiveness of our approach in handling complex scientific diagrams.
Groundth Truth Details. Due to the lack of publicly available datasets on scientific diagram segmen- lation, constructed custom dataset derived from the MMMU. This dataset focuses specifically on instances where the raw predictions GPT-4V (gpt-4-turbo) failed, thus representing challenging scenarios that test the robustness and precision of segmentation models. For our evaluation, collected a total of 100+ samples, with an even distribution across all scientific categories. Further examples and visual illustrations of these segmentation approaches provided in the Appendix B We prepared two distinct types of segmentation masks for each image:
a) Individual Segmentation Masks: These masks highly granular, with each individual visual element separated. For instance, the word "text" in a diagram is segmented into four distinct regions corresponding to each letter: 't', 'c', 'x', and 't', This detailed segmentation allows us precisely evaluate the accuracy of our split algorithm, gauging its ability to handle fine-grained details.
b) Grouped Segmentation Masks: In contrast, these grouped masks aggregate the visual elements into up to five cohesive semantic regions per image. This approach aims to assess how well the segmentation algorithm can identify and group related elements based on their semantic relationships.
Metries. To evaluate the performance of our method CoR and SAM2, we used the mean Intersection over Union (mloU) score as the primary metric, calculated for both the detailed and grouped segmentation masks. Unlike natural image segmentation tasks, our dataset lacks a one-to-one label mapping between ground truth and predicted regions. So, we use region-matching strategy: for each ground truth region MGt, we select the corresponding region M in the prediction with the largest area of overlap. The loU for this pair is calculated, and the process is repeated for all regions in the image. Once all individual IoU scores computed, the mloU for an image is determined by averaging the IoU values across all regions:
N
mloU
Σ
MMOT, n M.l/|Mor, U.l.
N
where N is the total number of regions in the ground truth mask, Mor, and M, represent the ground truth and predicted regions, respectively. For the i-th region, denotes the area of the region.
CoR Performs Better in Segmenting Scientific Diagrams than SAM2. We measure the perfor- mance of both our segmentation method and SAM2 using the mIoU, calculated separately for the individual and grouped mask versions: mloU (individual) quantifies the average overlap between the predicted masks and the ground truth in the Individual Segmentation setting, while mloU (group) evaluates the average overlap for the Grouped Segmentation setting. The results compiled in

---

Published as a conference paper at ICLR 2025
Table 2: Segmentation Accuracy the MMMU Dataset. presents the mean Intersection over Union (mloU) scores for both individual and grouped segmentation seltings.
Science
Tech and Engineering Energy
Method
Average
Bio.
Chem.
Geo.
Math
Physics
Arch.
CS
Elec.
Mater:
ME
mIOU (individual)
SAM2
36.8
47.7
30.1
39.9
56.4
39,9
44.7
52.5
30.4
22.7
37.4
42.2
CoR (Ours)
56.2
74.3
69.2
44.3
73.3
60.1
55.7
64.0
59.4
62.2
66.4
63.0
mIOU (group)
SAM2
26.2
31.2
18.6
19.5
20.0
28.7
30.0
29.5
38.2
36.0
28.8
CoR (Ours)
40.0
45.4
40.4
34.1
39.0
36.3
41.3
43.5
46.7
31.9
57.2
41.8
Table 2 where we present both scores for both our method and SAM2 across the different categories of scientific diagrams. Notably, our method outperforms SAM2 in both metrics, with a significant margin (20.8% the individual setting and 13.0% in the group setting). The numeric superiority highlights its fitness to scientific diagrams, making a valuable tool for enhancing the capabilities of VLMSs in scientific applications.
Qualitative Comparison between SAM2 and CoR. We supplemented our quantitative metrics with a qualitative visualization of the segmentation results in Figure [ (more in Appendix (B). focusing on direct visual comparisons between our method and SAM2. SAM2 struggled at providing accurate segmentation fine lines and small annotations depicted in blocks like equations s + 5/s + which are critical for fine- fine-grained understanding. Our method, based on traditional computer vision techniques, is good at identified and segmented these intricate details with a clean boundary. These examples underscore the importance of precise segmentation in scientific analyses, particularly in capturing subtle yet crucial elements that contribute to accurate scientific diagram understanding.
Individuual Group
KTF WT
Kiael) and 100m
[D3
Kal
Alr
CH
27+
(8) Original Image
(b) Ground (Human Labeli
(6) SAM2 Result
CoR Result (Ours)
Figure 5: Example of segmentation results on a scientific diagram. (a) Original image input. (b) Ground truth segmentations provided by human annotators. Two types of segmentations shown: individual segmentation, which aims capture all visual elements comprehensively. and group segmentation, which divides the image into up to five cohesive semantic regions. (c) Segmentation results generated by SAM2 (Research) (002)) (d) Results from the proposed Chain-of of Regions (CoR) approach. Each color in the masks represents the segment of unique region. We provide more examples Appendix B
5 CONCLUSION
In this work, we have developed comprehensive methodology to decompose regions and collect visual details in scientific diagrams using combination of traditional computer vision techniques and VLM. Our approach, which involves initializing, splitting, and merging regions, shows significant improvements in the analysis and interpretation of complex scientific imagery.
Limitations and Open Challenges. While our approach has demonstrated promising results, there are limitations and open challenges worthy to share: (a) Our library of structured detectors, crucial for identifying and parameterizing visual elements, is not exhaustive. The path to obtaining hidden parameters of structured visual elements inherently involves reverse-engineering their creation processes, which poses significant challenge. (b) Current DL methods also face limitations in this context. Unless they are specifically trained for task, DL methods struggle to generalize across the varied structures encountered in scientific diagrams.
In a nut shell, our work represents pioneering effort in combining traditional computer vision with modern machine learning techniques to tackle the problem of diagram analysis. Though the solutions we have developed are not universally perfect, we invite ongoing extensions to improve the applicability of our methods.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Guangji Bai. Zheng Chai, Chen Ling, Shiyu Wang, Jiaying Lu, Nan Zhang, Tingwei Shi, Ziyang Yu, Mengdan Zhu, Yifei Zhang, al. Beyond efficiency; A systematic survey of resource-efficient large language models. arXiv preprint arXiv:2401.00622 2024.
Gary Bradski. The openev library. Dr. Dobb's s Journal: Software Tools for the Professional Programmer, 25(1 11:111)222222 2000.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14455-144455 14465, 2024.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:4406.15884, 2024.
Google Cloud.
Multimodal generative ai search, 2024.
URL
https
//cloud.google / aww.aciimmiee---.e Learning/
multimodda--geeeattivv ai search Accessed: 2024-08-02.
PaddlePaddle Contributors. Paddlepaddle: An casy-to-use and efficient deep learning framework. wttn:/aaadeeaaccleeee.. /Paddle 2024. Commit ID; [Include specific
commit ID citing a particular version].
Michael Dorkenwald, Nimrod Barazani, Cees GM Snoek, and Yuki M Asano. Pin: Positional insert unlocks object localisation abilities in vims. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13548-13558, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL net vet/foos Wooouoriid CiNTTTTTTyy
Wan-Cyuad Fan, Yen-Chun Chen, Mengchen Liu. Lu Yuan, and Leonid Sigal. On pre-training of multimodal language models customized for chart understanding, arXiv preprint arXiv:2407.14506, 2024a.
Yue Fan, Lei Ding, Ching-Choe Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jic Yang, Yi Zhang, and Xin Eric Wang. Read anywhere pointed: Layout-aware gui screen reading with tree-of-lens grounding, 2024b. TRLLittps://arxist
wtwpg/////b///eee/ ..22266.19220 19263
Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, and Sifei Liu. Regiongpt: Towards region understanding vision language model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13796-13806, 2024.
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartlama: A multimodal I1m for chart understanding and generation. arXiv preprint arXiv:2311.16483 2023.
Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. Visual hallucinations of multi- modal large language models. arXiv preprint arXiv:2402.14683 2024.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22 22199222213 2022.
11

---

Published as a conference paper at ICLR 2025
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models:: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 16(1-2):1-21, 2024a.
Zhuowan Li, Bhavan Jasani, Peng Tang, and Shabnam Ghadar. Synthesize step-by-step: Tools templates and Ilms as data generators for reasoning-based chart vqa. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13613-13623, 2024b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26300 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403. 2024c.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chin Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and and--aaydyyaa compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yunze Man, Liang-Yau Yan Gui, and Yu-Xiong Wang. Situational awareness matters in 3d vision language reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13678 17788- 2024.
Cai Meng, Zhaoxi Li, Xiangzhi Bai, and Fugen Zhou. Are adjaceney matrix- based fast ellipse detection. IEEE Transactions Image Processing, 29:4406 4420, 2020.
Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401. 02384, 2024.
Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, al. Pivot: Iterative visual prompting elicits actionable 2024.
knowledge for vims. arXiv preprint
Frank Nielsen and Frank Nielsen. Hierarchical clustering. Introduction to HPC with MPI for Data Science, pp. 195-211, 2016
OpenAl. Gpt-4 technical report. 2023.
Nobuyuki Otsu et al. A threshold selection method from gray-level histograms. Automatica, 11 (285-296) 1975.
Wujian Peng. Sicheng Xie, Zuyao You. Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 13279-13288, 2024.
Meta Al Research. Segment anything model (sam2), 2024. URL https://at www..mta...mmmm com/ Accessed: 2024-09-28.
sam2/
Zeyi Sun, Ye Fang. Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong. Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. In Proceedings of the IEFF/CV Conference on Computer Vision and Pattern Recognition, pp. 13019-13029, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11 1805, 2023.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lm.. In Proceedings of the IÉEE/CVF Conference Computer Vision and Patlern Recognition, pp. 9568-9567 2024.
12

---

Published as a conference paper at ICLR 2025
Rafael Grompone Von Gioi, Jeremie Jakubowicz, Jean-Michel Morel, and Gregory Randall. Lsd: A fast line segment detector with a false detection control. IEEE transactions on pattern analysis and machine intelligence, 32(4):722-732, 2008.
Shaowei Wang, Lingling Zhang, Longji Zhu, Tao Qin, Kim-Hui Yap, Xinyu Zhang, and Jun Liu. Cog-dqa: Chain-of- of--gidingggggggoggooo learning with large language models for diagram question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1.3969-197999 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models, Advances in Neural Information Processing Systems, 15:24824-24837, 2022.
Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang, Jian Wu, and Philip Torr. Dettoolchain: A new prompting paradigm to unleash detection ability of milm. arXiv preprint arXiv:2403. /2488, 2024.
Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the power of referential comprehension for multi-modal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13838-13848, 2024.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang. Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07660 2023.
An Yan, Zhengyuan Yang. Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, al. List items one by one: A new data source and learning paradigm for multimodal lims. arXiv preprint arXiv:2404. 16375, 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unlcashes extraordinary visual grounding in gpt-4v, 2023. URL /atxps org at////b//33000 11441
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9536-9567, 2024b.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. The International Conference on Learning Representations, 2024a.
Jiarui Zhang, Jinyi Hu, Mahyar Khayatkhoei, Filip Ilievski, and Maosong Sun. Exploring perceptual limitation of multimodal large language models. arXiv preprint arXiv:2402.073386 2024b.
Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and Program-of-thougghe learning, arXiv preprint arXiv:2404. 16635, 2024c.
Wanrong Zhu, An Yan, Yujie Lu, Wenda Xu, Xin Wang, Miguel Eckstein, and William Yang Wang. Visualize before you write: imagination-guidet open-ended text generation. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 78 92, 2023.
13

---

Published as a conference paper at ICLR 2025
A METHODOLOGY DETAILS
A.1 VLM-ASSISTED STRUCTURE RECOGNITION
In this section, we use GPT-4V to preliminarily identify potential structures shapes within each main region. The input is prompted with the following instruction:
To better understand the details, you are shown only one part of the diagram one time. Now, please summarize +*all** the visual
clements in 2-3 words.
Please note that if there are multiple visual elements, use multiple
lines and each 1ine for one element.
Each line
with the coler of the visual element.
You should duplicate the line it there are multiple same visual elements.
An example of the input and output is shown in Figure 6 Detected keywords such as "arrow" and "circle" will trigger the corresponding structured splitting module, which we explain in more detail in Section A.2
Region Input
VLM Output
Region Input
VLM Output
Bluck di
Black artor
Olack circle
Black
Black cincle
Black artor
Black circlo
Black loop arrow
Figure 6: Two examples of region input and the corresponding output from GPT-4V.
A.2 STRUCTURED SPLIT
We have incorporated set of common structures typically encountered in the analysis of scien- tific diagrams. Below, we describe the detailed implementation of the structure.detectooz detector functions:
Ellipses. The el 1 lipse detector function is activated by the detection of keywords such as circle, oval, and ell psse during the Structure Recognition step (Appendix [A.T]. I. The imple- mentation leverages the open-source package cited in (Meng et al. 2020). The meta information extracted includes parameters such (Zcenter, Boenter: width, length, angle).
function initiated by the detection of keywords line, curve, and segment during the Structure Recognition step (Appendix I. This functionality is facilitated by the open-source package referenced in (Von Gioi et al. 20088. The meta information extracted includes parameters such
Segments. The vegment_detecte
(Istarts Nstarts Zends Yend, width).
Rectangles. The rect angle detector function is triggered by keywords rect, square, and bar identified in the Structure Recognition step (Appendix A.T]. The implementation process involves: a) Detecting all segments using the package cited in (Von Gioi et al., (008);; b) Merging collinear segments; c) Detecting segment quadruplets that form a convex shape. The meta information extracted includes (Irenter, Beenter, width, length, angle).
Arrows. The arrow_detected function activated by the keyword arrow, as detected in the Structure Recognition step (Appendix A.I). Given that prior steps confirm the presence of the line or curve, the
an arrow shape and segment_detect. direction is then determined by calculating the position of cv2 di stanceTrans form (M.i, DV2.DIST.L2. max () and comparing the two endpoints to identify which closer. The meta information extracted includes (Ista Usterts Fends Bend, width).
determines the trajectory
The structured library can be further expanded by incorporating additional shape templates, enhancing the robustness and utility of our system for complex diagram analysis.
14

---

Published as a conference paper at ICLR 2025
A.3 STRUCTURED MERGE
In this section, we detail the implementation of structured region merging, leveraging both existing tools and heuristic rules.
Captions. We employ mature OCR tools (Contributors) 2024) to detect and generate caption boxes within diagrams. Regions enclosed by these text boxes are presumed be related and are consequently merged into a unified region.
Dotted lines. Initially, collect line segments previously detected by the segment segment.detector .detector function, as introduced in Section A.2 Subsequently, we identify sequences of segments that meet the following criteria: a) The width and length of the boxes should be approximately equal (error <5%); b) The intervals between adjacent segment pairs should be similar (error <5%); c) The angular difference between the nearest pairs should be less than 20 degrees.
Continuing from the same initial line segments detected by segment. _dtreettddetettrrrrr we focus on those with either horizontal or vertical orientations. For hori- zontal line segments, we compile sequence of all y-axis positions, identifying the arithmetic mean with the largest gap and the longest contiguous segment. We then merge all horizontal lines whose y-axis values fall within the detected range. We apply analogous process for vertical line segments, merging those within the same x-axis range.
Background lines.
(s) Example of Detted Line Merge Figure 7: Two examples of dotted line merging and background line merging respectively. The original image faded, and the merged area is painted with different colors.
[b] Example of Background Line Merge
Final Prompts with Augmented Information
Information by -Rgiinn
[Original Question Choices]
Region
Structure Into
Semantic Info
Here
pre-extracted region information:
Region [A] overview: Circle labeled 'P1";
Girde at (115, 458): raclus 142
Chrouee lheeed 1P2"; Blue color the nock, Belld back anow pointing upwarts to
Region [8] overview: Ciroular node labeled "P2", Blue oolor for the pointing Arrow: (179, 456) to (386, 458)
radiaa
node; Solid
to "P2"
Anow dig
Region [C] ovenviaw: Circle labeled "P3" with one arrow pointing Inward and and arow pointing outward
Crrs aboled 'FS" with amww pointing insoeet nedt onw unow pointing cubward.
Answ- 222 437 49n
Anown (733 590 1255
Circle (798, 458): radius 142
Arrow: (520,
(729, 457) (505,
(753,
Orde 4922, 190 Packk 130
node bntweed "Py" with anow pointing
Region (D) ovenview: A node labeled 'P." with one anrow pointing Inward and one anow pointing outward.
Aemi: (8S,, th 20ng
Circle: (452, 793), Rediue 139
Arrow: (451, 521) (451, 720)
Arrow: (403,
515)
Region overview:
inside:
vedua Anris (155, 3410
petniing cac blue cirele from below
*P5" Arrow pointing towards the blue circle from below
Gircle at (465, 116j: maduus 142
Arrow: (455, 391) (455, 184)
Figure 8: Illustration how extracted information including structured information and semantic information is organized into an augmented prompt.
A.4 SEMANTIC INFORMATION EXTRACTION
In this section, we extract semantic information that includes the role and entity type of each merged region with the following prompt:
15

---

Published as a conference paper at ICLR 2025
To better understand the details, you shown only one part of the diagram one time. Now please **only** summarize the highlighted visual elements in short phrases.
A.5 CHAIN-OF-THOUGIT PROMPT
In this section, we include the prompt used in the Few-shot Chain-of-Thought baseline: [Original Question
Choices]
# Please summarize regions in the figure before answer the quostion. Here is the example of extracted region information:
Region [A]: Circle labeled "P1";
Region [B]: Circular node labeled "P2"; Blue coler for the node; Solid black pointing upwards to "P2"
Region [C]: Circle labeled "P3" with one arrow pointing inward and one arrow pointing outward
Region [D]: A node labeled "$P_4$" with arrow pointing inward and one arrow pointing outward.
12
13 Region [E]: Blue circle with black border; Text inside: "P5"; Arrow pointing towards the blue circle from below
B EXTRA VISUAL EXAMPLES
Segment: Individual Elements
Segment: Grouped Regions
Original Image
Ground Truth (Human Label)
CoR Prompts
Ragion ovardea: Word "start blick text: Black pointing Faint concentric circles
SAM2 Result
Annow: 221) (206, 221)
Region [B] ovarview: Two circles labaled "G'"
and G8") Arrowa connecting G1 to G6:
After Region Split
After Region Merge
Arowheads at both ends between G1 and G6
Circle at (287, 221): racius 153
Circle (297,
186
Ellpse
155, height
CoR Result (Ours)
Arow: 222y
(485,
Text 1622, 195) "o"
Figure 9: Example of CoR prompts and examples of segmentation results on scientific diagram including: original image input, ground truth segmentations provided by human annotators. and segmentation results generated by SAM2 (Research) 2024) and the proposed Cf (CoR) approach.
16

---

Published as a conference paper at ICLR 2025
Segment: Individual Elements
Segment: Grouped Regions
Original Image
Open
Syocistt
Opeept
Ground Truth (Human Label)
Jmc 0n
I--A0m
Fig.
Prompts
Regicn [A] ovandiew: Measurement tiee marked et: Sanudged noo or symbel marked 'P1". Dotted tectrr on the background
Raclangular: xmm 56, ymin 247, amax 357, ymax
661
Ragion [8] ovensew: Labeled "Smooth pin': Dameter indicated as 0.5 om; Vertical the representing the pin Text 1350, 143): "smooth pice
Text 1354, 1871: 'd-5 cm
Region (C) evervies Two vertical lines, possibby representing measurements (dimens ons; Label 'Fig. P8.52' at the bottom center.
After Region Split
After Region Merge
Text (641, 684): 160 m²
Text (467, 774): "fg. 96.52
CaR Result (Ours)
Opea
Ragion ovendew: Bert pipe; Dimsnssion labeled '80 mm: Smooth p surface indioated
Text 1861, 380): mm
Pagion cveevien: Bant labwied '30 cm Arrow ind
direction labeled "Open
Text [851,
23m
(Oriittad arow
better (lustration)
Figure 10: Example of CoR prompts examples segmentation results on a scientific diagram including: original image input, ground truth segmentations provided by human annotators, and segmentation results generated by SAM2 (Research) 2024) and the proposed Chain-of of-Regions (CoR) approach.
Segment: Individual Elements
Segment: Grouped Regions
Original Image
Jun
Kin+
Cli
Ri
Cdll
Ground Truth (Human Label)
RT 200+
XP 33V
7+7
1::5
CoR Prompts
Pagion [4] cvervient Ciicke with crees Indicating
suamiin junctiont. Two input signals:
V: Output from the junction:
Text
SAM2 Result
"lg"
-Circle radius 163)
(288. (414, 161)
Ragion ovensieu:
After Region Split
After Region Merge
expression
3)'
Reclangular. bar
10x>
06,
CoR Result (Ours)
ymax 1861, "W 1960, 2013 sal
Figure 11: Example of CoR prompts and examples of segmentation results scientific diagram including: original image input, ground truth segmentations provided by human annotators, and segmentation results generated by SAM2 (Research) 2024) and the proposed Chain-of-Regions (CoR) approach.
17


Reviews:
[{"review_id": "bKnGV858Ne", "reviewer": "Reviewer_sLjr", "summary": "This paper developed a new method called \"Chain-of-Regions.\" This method is to help language models to accurately understand the visual information present in a figure such as scientific graphs or diagrams. While general models like GPT have great capabilities to understand images accurately, some gaps need to be filled. The novelty of this paper lies in that the authors adopted traditional CV techniques (not deep learning models) to decompose the image before feeding the broken-down parts into the visual language model.\n\nThe proposed CoR method involves a three-step process. The first step (Region Initialization) is to break down regions using connected components analysis offered by OpenCV. This initially identified regions further analyzed in the second step (Region Splitting). The second step further identifies detailed components in each region by checking whether common geometric shapes exist or unstructured shapes that can be identified from the liquid filling analogy. Lastly, the third step (Region Merging) finally checks how many regions can be practically processed in order to make it computationally efficient enough.\n\nBased on the proposed CoR method, the authors report significant improvement in performance in region segmentation. The authors also conducted various ablation studies that confirms each step's contribution to the overall performance enhancement. With regard to implementation, the refreshing point is that this CoR method can be run on CPU only.", "questions": "1) How does the proposed method fare on parsing out real-world scientific diagrams?\n\n2) Did you consider using other operators or functionalities of OpenCV as alternatives to the connected components analysis and the shape detection algorithm?\n\n3) The structured splitting part of Step 2 involves identifying commonly used geometric shapes. What shapes are included in the library, and how comprehensive are they in your assessment?\n\n4) All the performance reported in the paper is about accuracy (correct me if I am wrong). However, the authors push cost and computation efficiency (CPU only) as the main strengths of their approach. Do you have any hint of comparisons about those computational complexities and how they scale?\n\n5) As I mentioned in the weaknesses, I am particularly interested in learning more about the interface between OpenCV and VLM. How did you ensure the regions detected by OpenCV are correctly interpreted by the VLM downstream? Were there any possibilities of conflict between the two modules?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The novel and refreshing contribution of this paper is that employing old computer vision techniques in a smart way can be very helpful with modern visual language models. Inspired by this paper, future studies may look into such possibilities to replace certain parts of the language model pipeline.\n\n2) The nice thing about the CoR method is that it can be solely run on CPUs (because it only uses OpenCV's connected component analysis and shape detection algorithm), which means it's very cost-effective and probably fast too.\n\n3) The results look solid, as shown by the extensive ablation studies.\n\n4) Another nice property of this method is that it can be used in a Plug-and-Play fashion with the existing visual language model pipeline. This is actually a good property because it can actually have some practical impact in this ever-growing LLM era. The proposed method's versatility in identifying different shapes also makes it easily adopted in the operating pipeline as a new component.\n\n5) Lastly, this proposed method is clearly a white-box approach, which is transparent in showing why the regions were split in a certain way.", "weaknesses": "1) One thing that I find a bit short is the theoretical justification of why the authors made certain technical and/or methodological choices. I think the implementation and the practical implications are strong, but the authors fall short of explaining why their approach works better. The three steps of Region Initialization, Region Splitting, and Region Merging make sense to me, but having a more well-written formal justification of these steps would make the paper rock solid.\n\n2) While the authors compare the CoR method with SAM2, I wonder how the CoR method performs compared to other specialized models for processing scientific diagrams. While impressive, the results would be more robust if the authors could present a bit more head-to-head comparisons with competing models and/or approaches.\n\n3) The results are unanimously great, which is good of course, but it also makes me wonder what is the boundary condition and where this approach fails. Currently, the results look just too good. Knowing when and where a model fails is very helpful for future studies to build upon. In addition, the limitations that the authors acknowledge at the end are just nominal---there may be some other geometric shapes in our library. I want to hear about the real limits of this approach.\n\n4) Compared to Steps 1 and 2, I think Step 3 (Region Merging) needs a bit more clarification. It involves budget parameter B, but I don't find much justification or rigorous discussion on the choice of B. Discussing at least the trade-off between a large and a small B would be nice.\n\n5) Since this paper's core novelty lies in the combination of a traditional method with a modern VLM, the authors need to explain better how the output of OpenCV and CoR is fed into and/or interacted with the VLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKCtrdmyI2", "reviewer": "Reviewer_WY7p", "summary": "This paper presents a method for scientific diagram understanding that combines traditional CV methods (i.e., region initialization, splitting, and merging) with VLM prompting. The approach is evaluated on two both MMMU scientific diagrams and a small-scale diagram segmentation dataset. Results indicate the superiority of the proposed method.", "questions": "1. How many diagrams are there exactly in the dataset used in experiment 2? The authors wrote \"100+\", but this seems very unrigorous... In addition, how were these diagrams selected? I noticed all examples shown for exp 2 are in grayscale, which leads me to wonder if they were specifically chosen to play to the strengths of the proposed method that is largely heuristics-based.\n2. Can the authors provide a decomposition of the number of charts for each category in experiment 1 for the sake of clarity? \n3. What is a rough cost comparison of the different prompting approaches in Experiment 1? If CoR is very expensive compared to other approaches, would some form of inference-time scaling (e.g., generating multiple responses and using majority voting) improve baseline method performance? I'm not sure if this is a valid thing to do, so please feel free to argue if this is a bad idea.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I appreciate how this work recognizes that a key challenge of diagram understanding is parsing the details and proposes a method combining classical CV methods to complement VLMs.\n- The authors benchmarked a wide array of prompting strategies in their first experiment.\n- The overall narrative is easy to follow.", "weaknesses": "I want to preface this by saying that I am a relatively junior reviewer, so please take my comments with a grain of salt.\n- The authors mentioned they developed some structured detectors to identify and parametrize visual elements in diagrams. The performance of the method also seems to depend on the diagram having homogeneous colors and structured patterns. This raises questions about the generalizability of the approach. In experiment 1, the authors only evaluated the method on MMMU alone. Therefore, I wonder how this partially heuristics-based method would fare on other datasets. Some extra experiments might be helpful for demonstrating the power of the method.\n- Some crucial details about Experiment 2 (on segmentation) seem to be missing. How many diagrams are there exactly in the dataset used in experiment 2? The authors wrote \"100+\", but this seems very unrigorous... In addition, how were these diagrams selected? I noticed all examples shown for exp 2 are in grayscale, which leads me to wonder if they were specifically chosen to play to the strengths of the proposed method that is largely heuristics-based. In addition, only SAM2 was compared against CoR. Perhaps it would be helpful to include some additional baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3WjNWMBkm", "reviewer": "Reviewer_dTRw", "summary": "This paper introduces the Chain-of-Regions (CoR) technique to enhance VLM's fine-grained recognition and reasoning capabilities for scientific charts. It demonstrates performance across disciplines including mathematics, architecture, and electronics. This paper combines established computer vision techniques with emerging topics in cross-modal understanding. I appreciate the approach of combining traditional computer vision methods to assist in enhancing the model’s performance, and indeed, the work shows a thorough consideration of various experimental details. The following  are some suggestions that may help the authors improve this work.\nTraditional computer vision techniques, including the use of OpenCV, tend to be relatively sensitive to parameters. Yet, in the paper, it seems that authors have not analyzed the sensitivity of these parameters or how the understanding of diagram may vary across different scenarios.\nAdditionally, the rule-based reverse engineering appears to lack sufficient robustness—there has not been a comprehensive investigation to form the systematic theoretical framework for pixel recognition, segmentation, extraction, and integration. I suggest strengthening the discussion on this point.\nNotably, sequential segmentation and rule-based combination are insufficient, as they may disrupt semantic context. I suggest that authors incorporate innovative algorithms in future work to further explore the interrelationships between different pixel regions.\nIn terms of writing, I suggest that the authors provide more pseudocode for the operations instead of directly presenting OpenCV code.\nThe parameter “Budget B” may also introduce uncertainty in the interpretation of subsequent diagram, but the authors have not provided much discussion on this.\nIn the Region Input of Appendix A1, we obtained the Output of different VLMs. It seems that this output is scanned from left to right. However, if the rotation angles of these images differ, what would the result be in that case?", "questions": "Traditional computer vision techniques, including the use of OpenCV, tend to be relatively sensitive to parameters. Yet, in the paper, it seems that authors have not analyzed the sensitivity of these parameters or how the understanding of diagram may vary across different scenarios.\nAdditionally, the rule-based reverse engineering appears to lack sufficient robustness—there has not been a comprehensive investigation to form the systematic theoretical framework for pixel recognition, segmentation, extraction, and integration. I suggest strengthening the discussion on this point.\nNotably, sequential segmentation and rule-based combination are insufficient, as they may disrupt semantic context. I suggest that authors incorporate innovative algorithms in future work to further explore the interrelationships between different pixel regions.\nIn terms of writing, I suggest that the authors provide more pseudocode for the operations instead of directly presenting OpenCV code.\nThe parameter “Budget B” may also introduce uncertainty in the interpretation of subsequent diagram, but the authors have not provided much discussion on this.\nIn the Region Input of Appendix A1, we obtained the Output of different VLMs. It seems that this output is scanned from left to right. However, if the rotation angles of these images differ, what would the result be in that case?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper combines established computer vision techniques with emerging topics in cross-modal understanding. I appreciate the approach of combining traditional computer vision methods to assist in enhancing the model’s performance, and indeed, the work shows a thorough consideration of various experimental details.", "weaknesses": "The limitations of this paper lie in the potential impact of parameter sensitivity on the final interpretation, as well as whether the overall framework is adequately supported by methodological foundations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gsrdo1r1gC", "reviewer": "Reviewer_fCtq", "summary": "This paper presents Chain-of-Region, a prompting method that enhances the capabilities of VLMs in scientific diagram analysis. This is achieved by cropping out regions of the diagram with traditional computer vision techniques based on heuristics and incorporating the extracted information into a master query. The effectiveness of this method is verified by evaluating the QA accuracy and the segmentation results on the MMMU dataset.", "questions": "1. Why are background masks included in the region initialization? They appear to contain minimal information in the provided examples. Also, would they be regarded as the “main regions with large area size” in the VLM-assisted structure recognition?\n2. How beneficial is the unstructured split and merge approach for analyzing scientific diagrams? Illustrations requiring this method are often considered single entities, as shown with the server in Fig. 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Practical methodology. This paper proposes a practical method for leveraging VLMs in scientific diagram analysis, allowing for plug-and-play generalization across models.\n2. Significant performance gain. The method shows notable performance gains when applied to state-of-the-art models like GPT-4o.\n3. Clear presentation. The paper is generally well-written, with clear figures that aid comprehension.", "weaknesses": "1. Limited generalizability. The method contains various heuristics tailored for scientific diagrams, limiting its applicability to more complex images. Extending this method for segmenting such images can be very challenging, if not impossible.\n2. Unclear intuition. The rationale for how adding information from regions enhances QA task performance is not clear. While the example in Fig. 1 demonstrates how prompts lead the VLM to predict values based on the numbers in the prompts, I can’t see how they can aid the reasoning of various types of diagrams, such as the node-link diagram in Fig. 2. Providing more examples of the QA results and analyzing them can help better clarify this.\n3. Unfair evaluation. The evaluation in Sec. 4.2 appears biased, as ground truth annotations favor the proposed method. The human labels only include foreground masks, penalizing methods that incorporate background masks, even when the cropped regions are identical (see the masks for K(s+1)/s(s+2)(s+3) as an example). A more appropriate evaluation method would use the area of the ground truth $|M_{GT_{i}}|$ as the denominator in the mIoU calculation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)