InfoDet: A Dataset for Infographic Element Detection
Authors: Jiangning Zhu
Yuxing Zhou
Zheng Wang
Juntao Yao
Yima Gu
Yuhui Yuan
Shixia Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Infographics
visual reasoning
grounded chain-of-thought
object detection
dataset
Primary area: datasets and benchmarks
TL;DR: We build InfoDet, a dataset for infographic element detection

Abstract: Given the central role of charts in scientific, business, and communication contexts, enhancing the chart understanding capabilities of vision-language models (VLMs) has become increasingly critical. A key limitation of existing VLMs lies in their inaccurate visual grounding of infographic elements, including charts and human-recognizable objects (HROs) such as icons and images. However, chart understanding often requires identifying relevant elements and reasoning over them. To address this limitation, we introduce InfoDet, a dataset designed to support the development of accurate object detection models for charts and HROs in infographics. It contains 11,264 real and 90,000 synthetic infographics, with over 14 million bounding box annotations. These annotations are created by combining the model-in-the-loop and programmatic methods. We demonstrate the usefulness of InfoDet through three applications: 1) constructing a Thinking-with-Boxes scheme to boost the chart understanding performance of VLMs, 2) comparing existing object detection models, and 3) applying the developed detection model to document layout and UI element detection.

Published as a conference paper at ICLR 2026
INFODET: A DATASET FOR INFOGRAPHIC ELEMENT DETECTION
Jiangning Zhu', Yuxing Zhou', Zheng Wang's Juntao Juntao Yao
Yima Gu', Yuhui Yuan Shixia Liu BNRist, Tsinghua University "Cana CORE
11,264 real infographics
InfoDet
14,227,680 annotations
90,000 synthetic infographics
mini
04 mini-PoT
DINO-X T-Rex2
DETR with Co-DETR with InfoDet
Without InfoDet
With InfoDet
04-minixCoT
04-mini-Grounden CoT
Re ared Ado.
65
ChartQAPro
80
InfoDet
54
Rico
61
DocGenome
+1.7
+49.6
+1.3
62
52-
79
77
Chart Understanding via Grounded Chain-of Thought
Object Detection Model Evaluation
Graphic Layout Detection
Figure 1: Key contributions: 1) An open-source dataset InfoDet. 2) Improvements on chart understanding, infographic element detection, and graphic layout detection.
ABSTRACT
Given the central role of charts scientific, business, and communication con- texts, enhancing the chart understanding capabilities of vision-language models (VLMs) has become increasingly critical. A key limitation of existing VLMMs lies in their inaccurate visual grounding of infographic elements, including charts and human-recognizable objects (HROs) such as icons and images. However, chart understanding often requires identifying relevant elements and reasoning over them. To address this limitation, we introduce InfoDet, a dataset designed to support the development of accurate object detection models for charts and HROs in info- graphics. It contains 11, 264 real and 90, 000 synthetic infographics, with over 14 million bounding box annotations. These annotations are created by combining the model-in-the-loop and programmatic methods. We demonstrate the usefulness InfoDet through three applications: 1) constructing a Thinking-we -ithh -ixt- scheme to boost the chart understanding performance of VLMs, 2) comparing existing object detection models, and 3) applying the developed detection model to document layout and UI element detection.
Code:
Data & Dataset Card: hwwp://wwwmmgggiiooooooooooooooo
INTRODUCTION
Charts are a fundamental medium for conveying data-driven insights across scientific, business, and communication domains. Consequently, improving vision-language models (VLMs) for chart understanding has become increasingly critical, driving significant advances in understanding plain charts (Huang et al. 2024 -minimal combinations of texts and charts. In practice, however, charts are often combined with icons and images of real-world objects, known as human-recognizahly objects (HROs) (Borkin et al., (016), to create infographics. By thoughtfully arranging texts, charts, and HROs, infographics transform abstract data into accessible insights through engaging visual
"Corresponding author.

---

Published as a conference paper at ICLR 2026
designs. Due to their widespread use, automated infographic understanding using VLMs has become increasingly important for real-world applications such as news analysis (Chen & Chang 2023) and accessible content generation (Hohenwalde & Hazim 2025). However, previous studies Masry et al., 2025 Li et al., 2025a) have identified a key limitation of existing VLMs: inaccurate visual grounding of infographic elements, which hinders the ability to associate the elements with the underlying data. This highlights the need for robust object detection models to support visual grounding and enhance chart understanding. Although considerable progress has been made in text detection (Long et al. 2021; Du et al.) 2020), detecting charts and HROs key elements linking abstract data to human perception remains relatively underexplored.
Compared to natural scenes, element detection in infographics presents challenges for two reasons. First, infographic elements exhibit high intra-class variance. Charts vary widely in type, layout, and visual design, and HROs appear in diverse styles, spanning from realistic depictions to abstract representations of real-world objects. Second, the visual interplay between charts and HROs often results in ambiguous boundaries, making it difficult to distinguish one element from another in context. To effectively handle the highly varied infographic elements with ambiguous boundaries, the detection model needs to learn from diverse set of infographics with accurate annotations. Existing datasets, however, primarily focus on plain charts without HROs (Battle et al. 2018) Deng et al. 2023), failing capture the complexity of infographies. Borkin al. (2016) have taken the first step in building a dataset with rich annotations, but their dataset is limited in scale, comprising only 393 samples due to the labor-intensive manual annotation process. To advance element detection in infographics, large-scale dataset of diverse infographics with comprehensive annotations is required.
To fill this gap, we introduce InfoDet, a dataset for infographic element detection. It comprises a diverse collection of infographics from two sources: 1) real infographics collected from 10 online plat- forms, such as Visual Capitalist and (Statistal, and 2) synthetic infographies programmatically created from 1, 072 design templates. To effectively annotate the infographics. we combine programmatic and iooel--n-the-llop (Kirillov et al. 2023) methods. For the synthetic infographics, we program- matically derive the bounding boxes. For the real infographics, we co-develop an object detection model and the annotation process, allowing the model and the annotations to iteratively enhance each other. Specifically, we use the annotated synthetic infographics to fine-tune InternImage-base object detection model (Wang et al. 2023), which is then employed to generate annotations for all real infographics. The generated annotations multiple rounds of refinement. In each round, expert feedback is utilized to enhance the annotation quality and refine the model, thereby progressively improving its accuracy. In total, InfoDet contains 1 11,264 real and 90,000 synthetic infographics, along with 14,227,680 bounding box annotations of texts, charts, HROs, and finer-grained sub-elements such as bars, axes, and legends. Table 6 in the Appendix provides a statistical comparison of InfoDet with existing chart datasets.
reviewed and corrected by the experts through
We demonstrate the usefulness of InfoDet through three applications (Fig. D. First, we propose Thinking-wind with-Boxes scheme that performs grounded chain-of-thoughh reasoning over elements. This grounded reasoning considerably improves the performance of OpenAl o4-mini on the challenging ChartQAPro benchmark (Masry et al.] (025). Second, we compare the performance of the state- of-the-art object detection models. The results show that the best-performing foundation models for object detection (e.g., DINO-X (Ren et al. 2024)) still struggle to accurately detect infographic elements, whereas fine-tuning traditional object detection models (e.g.. Faster R-CNN (Ren et al.. 2015)) with InfoDet achieves improved performance. These findings highlight the importance of sufficient high-quality training data for chart and HRO detection. Third, we apply our InternImage- based object detection model to -fo-oi- graphic layout detection tasks, including document layout and UI element detection, demonstrating its generalization capability across broader domains.
The main contributions of this work threefold:
A large-scale dataset for infographic element detection with 101, 264 annotated infographics.
An Internimage-based model for detecting charts and HROs in infographics.
Three applications that show InfoDet's usefulness in chart understanding and object detection.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Based on the presence of HROs, chart datasets with element annotations can be classified into two categories: datasets of plain charts and datasets of infographic charts.
Plain charts present data in minimal manner using texts and charts. Some datasets consist of programmatically created charts. FigureQA (Kahou al. 2018) comprises 100, 000 charts created from randomly generated data using Bokeh] However, relying on randomly generated data limits real- world representativeness. To address this, Methani et al. (2020) use crawled data to create 224,37 377 charts by randomly combining design parameters such as marker and line styles. Other datasets are constructed by collecting charts from existing literature or online platforms. VG-DCU (Dou et al.) 2024) consists of 15, 197 SVG-based charts, from which bounding box annotations by analyzing the SVG elements. Vislmages (Deng et al., 2023) is constructed by gathering 12, 267 images from IEEE VIS publications and manually annotating 35, 096 charts within them. While these datasets facilitate object detection model training for plain charts, such models often struggle with the widely used infographics, where diverse HROS and their interplay with the charts introduce significant variability.
extracted
To better support the analysis of infographic designs, Borkin et al. 2016) pioneered the creation of an infographic dataset with rich annotations. They utilize an existing database of real infographics and manually annotate the polygons of their elements. However, this dataset is limited in scale, comprising only 393 samples due to the labor-intensive manual annotation process. As result, this dataset is unsuitable for training object detection models that require strong generalization. In contrast, InfoDet combines a model-in-the-loop annotation method for real infographics and programmatic annotation method for synthetic infographies, resulting in 101, 264 annotated infographics that effectively support object detection model development.
INFODET CONSTRUCTION METHOD
Fig. provides an overview of the dataset construction pipeline, which includes main steps: infographic collection and infographic annotation.
3.1 INFOGRAPIII COLLECTION
Previous studies (Zhu-Tia Tian et al. 2020 Zhu et al. 2025) have highlighted the complementary benefits of real and synthetic data: the former captures authentic design practices, while the latter offers controlled variation and scalability for robust training and evaluation. Informed by this finding,
Infographic Collection
Infographic Annotation
Sodel--n-tthe-loo
statista
Experts
Feedbock
Collection tion from platforms
Deduplication Content verification
11,264
WORK
real Infographics
Detection model
Experts summarize
14, 227,600 annotations
Synthetic Infographic Creation
Annotated infographics
Monthly Energy Usage in Lirban and Rural Areas
van
31M data tables
C
1,072 templates
90,000 synthetic infographics
845K Icons
Figure 2: The construction pipeline for the InfoDet dataset.
3

---

Published as a conference paper at ICLR 2026
collect infographics from two sources to balance authenticity, diversity, and scalability: 1) real infographics from chart-rich online platforms, and 2) synthetic infographics created programmatically.
Real infographic collection collect infographics from chart-rich online platforms that permit research use, such as Visual Capitalist and SStatista] The complete list is provided in Appendix [B] To enhance the quality of the infographics, we implement two-step filtering process: deduplication and content verification. In deduplication. we remove infographics that exhibit high CLIP similarity (Rad- ford et al. 2021) (2 0.9) and low perceptual hashing distance @Jain et al. 2019) (< 2) relative to other infographics. In content verification, we prompt GPT-40 mini to identify and remove images that are blurry, lack charts or HROs, are photographs instead of graphic designs, or contain personal or sensitive content. After filtering, the collection is refined to 11, 264 high-quality infographics.
Synthetic infographic creation We employ a template- based method (Li et al. 2025b) create synthetic infographics. This method utilizes 1, 072 design templates derived from representative real infographics. Each template specifies: 1) the presence and relative positions of charts, texts, and HROs, and 2) the type and visual style of the charts. An infographic is created by filling the template with: 1) data tables for chart creation, 2) descriptive texts, and 3) selected HROs. To ensure diversity, we sample data tables from VizNet (Hu et al. 2019), a large-scale dataset containing over 31 million tables and associated metadata. Charts are created from the sampled data tables as specified by the template. Descriptive texts for the charts are generated using GPT-4o mini. HROs with the highest CL.IP similarity to the descriptive texts are selected from the IconQA dataset (Lu et al. 2021). } which contains over 645K icons. Using this template-based method, we generate 90, 000 synthetic infographics. Example templates and infographics generated from them are provided in Appendix C
3.2 INFOGRAPHIC ANNOTATION
Given the differences in collecting real and synthetic infographics, we adopt two annotation methods: a programmatic method for synthetic infographics and a model-in-the-loop method for real ones.
Programmatic synthetic infographic annotation Synthetic infographic annotations are program- matically generated with a parser integrated into the infographic generation process. This parser extracts bounding boxes for texts, charts, and HROs from the corresponding SVG file, which encodes the visual and structural details of the infographic. Additionally, the parser leverages information from the design template to classify charts and HROs into subcategories: charts are categorized into 75 distinct types, while HROs are labeled as either data-related or theme-related objects. The complete list of chart types is provided in Appendix D
Model-in-the-lopp real infographic annotation To reduce human labor in the annotation, we aim to leverage object detection models for assistance. However, there is an absence of specialized detection models for charts and HROs. To address this, we employ a model-in-the-loog annotation method (Kirilovv et al., 2023). This method co-develops object detection model and the annotation process, allowing the model and the annotations to iteratively enhance each other. Specifically, using the annotated synthetic infographics, we build object detection model by fine-tuning Internlmage- L. (Wang et al. (023) along with the DINO (Zhang et al. 2023) detector. This fine-tuned model is then employed to generate annotations for all real infographics. However, since the synthetic infographics do not fully represent the diversity of all infographics, the fine-tunee object detection model is prone to errors. To mitigate this, we conduct multiple rounds of annotation refinement and model enhancement with the experts. In each round, the experts review and correct the auto- sutto-generaed annotations, and the feedback is used to further fine-tune the model, progressively improving its accuracy. More details on the expert background and the refinement rounds are provided in Appendix E At the end of the refinement process, we randomly sample 1, 250 infographics to evaluate the quality of the generated annotations. Results show that the generated annotations achieve a precision of 93.9% and recall of 96.7%, comparable to those of widely used object detection datasets, such as COCO (Lin et al. 2014) (83.0% recall and 71.9% precision) and Objects365 (Shao et al. 2019) (92.0% recall and 91.7% precision), as reported by (Sha et al. (2019). We analyze the remaining model errors and discuss the dataset scope in Appendix F
3.3 STATISTICS AND DATASET ANALYSIS
InfoDet contains 101,264 infographics, including 11,264 real and 90,000 synthetic infographics. To complement the dataset with text annotations, we use the widely adopted OCR model PP-OCRv4 (Du
4

---

Published as a conference paper at ICLR 2026
al. 2020) annotate all real infographies and extract text annotations from the generation process for synthetic infographics. For completeness, we also extract mark-level annotations 26 categories of chart sub-elements, such bars, axes, and legends. The list of sub-element categories is provided in Appendix G In addition, we extract segmentation masks for the charts, HROs, and chart sub-elements using SAM M (Kirillov et al. (023) support finer-grained analysis. Across these infographics, annotate a total of 7,284,8% 892 text elements (each corresponding to line of text), 310,299 charts, 1,080,598 HROs, and 5,551,891 sub-elements. The detailed statistics
provided in Appendix D Beyond the basic statistics, we have verified that our dataset exhibits high diversity, no harmful bias, and high fidelity of synthetic infographics (See Appendix HH. To ensure consistent evaluation, we split InfoDet into training set of 96, 264 infographics and test set of 5, 000 infographics, while maintaining the same proportion real and synthetic infographics both sets.
4 EXPERIMENTS
In this section, we first construct a Thinking-wiid with-Boxes scheme to enhance the performance of the latest VLMs. We then evaluate the performance of existing object detection models. Finally, we apply the Internlmage-base based object detection model to graphic layout detection tasks.
4.1 THINKING-WITI-BOEES VIA GROUNDED CHAIN-OF-THOOUGN
Latest VLMs, such OpenAl's s 03 04-mini demonstrate chain-of-thought reasoning capability with images through seamless image manipulations, including automatic zooming and cropping. Considering that chart understanding requires more complex, fine-grained visual reasoning over the elements within infographic images (Lin et al. (2025), we construct a Thinking with-Boxee scheme to enhance VLMs by explicitly providing grounded annotations of texts, charts, and HROs along with additional layered infographic images. The bounding boxes are predicted using our infographic- oriented object detection model and OCR model. With this scheme, output reasoning trajectories over the grounded regions, referred to as grounded chain-of-thought (grounded CoT), which guide the model think step-by-step before achieving the final answer.
prompt the VLMs
4.1.1 GROUNDED CHAIN-OF-THOOCGHT PROMPTING
As shown in Fig. BBB we provide the VLM with detected elements in two modalities-visual prompts, by overlaying boxes the infographic image, and textual descriptions of each element- study the reasoning preferences of the evaluated VLMs.
For the visual prompts, we overlay bounding boxes top of the infographics, each labeled with an alphabetical ID. To improve clarity, the bounding boxes are rendered contrastive colors against the background, and the ID labels are placed minimize overlap. However, even with these measures,
Bounding Boxes
Textual Descriptions
AARC
+
E
ABC
ABC
@VVisual Layers
Original Image
Detection model
HRO
HRO
Visual, Layers
E
ABC
Text
Text
Textual, 2 Layers
Annotated Images
ABC
Overlay
Texuull Layer (b)
(a)
Figure 3: The Thinking-wind with Boxes scheme: (a) the charts, HROs, and texts detected and overlaid onto the original image to create annotated images with grounded elements; (b) the input of the grounded chain-of-thouge method (B, and its ablated variants (B2, B3, B4).
5

---

Published as a conference paper at ICLR 2026
Table
Performance of ol, o3, and o4-mini with different prompting methods. The best one is bold.
04-minni
Chart Group
Grounded CoT (ours)
Grounded CoI (ous)
Grounded CoI (ours)
Direct
CoT
PoT
Direct
CoT
Pot
Direct
CoT
PoT
Plain, Single
$7.8
57.8
56.1
60.1
56.8
$7.7
57.5
57.2
58.1
57.9
55.3
60.6
Plain, Multiple
63.7
65.1 1
62.2
65.4
62.8
61.0
58.8
63.4
66.7
66.1
62.3
66.9
Infographic, Single
60.9
59.5
64.2
67.5
Infographic, Multiple
66.0
67.6
66.8
71.9
66.0
64.9
64.2
68.8
70.6
69.2
64.7
72.5
Overall
61.4
61.9
60.0
64.1
60.0
59.5
62.5
59.7
64.9
overlap between bounding boxes remains inevitable in regions with dense texts and HROs. To mitigate this, we propose to separate the visual prompts into two layers: one containing charts and HROs, and the other containing texts. We also provide textual description of each element to ease the challenge of simultaneously locating and interpreting their content. Please refer to Appendix I.T for detailed prompts and comparison of visual prompts rendered in one versus two layers.
4.1.2 EXPERIMENTAL SETUP
We evaluate the chart understanding capability of VLMs using ChartQAPro (Masry et al., (025). ChartQA (Masry et al. 2022), and PlotQA (Methani et al.) 2020). To better analyze the performance of our method, we manually categorize them into four groups based on two criteria: whether the charts are plain or infographic, and whether there are single or multiple charts. We assess three ofate-of-thh-ar VLMs: OpenAl's ol, oo] and jo4-mini For each VLM, we compare our method against three widely used baseline prompting methods: 1) Direct prompting with the chart image and the question, 2) Chain-of-Thoughe (Wei et al., 2022) (Co'T), which prompts the model to reason step-by-step for the provided image and question, and 3) Program-of-Thougge (Chen et al., 2023) (PoTT), which prompts the model to generate a Python code that prints the final answer. The performance is measured using the enhanced relaxed accuracy (Masry al. (2025). Please refer to Appendix I. for the detailed prompts and the enhanced relaxed accuracy implementation.
4.1.3 RESULTS AND ANALYSIS
The ChartQAPro results shown here. The ChartQA and PlotQA results are in Appendix J.1
Effectiveness of grounded CoT prompting As shown in Table prompting the latest VLMs to think step-by-step or write Python code does not significantly improve their performance. We attribute this the reasoning-centre design of the VLMs. which inherently reduces the dependence on explicit prompts for step-by-step reasoning. In contrast, our method enhances chart understanding performance by providing grounded infographic elements. In particular, our method performs comparably on plain, single charts and shows better performance on infographic charts and images with multiple charts, leading to improved overall performance. To assess the robustness of the observed gains, include an additional set of runs 03 and present the results in Appendix J.2] As shown in Fig. the grounded annotations of elements effectively guide the VLM to reason step-by-ste bt-step and arrive at the correct answer. We provide more examples in Appendix [33 Despite its strong visual reasoning capability, o3 encounters instruction-following issues, resulting in slightly lower performance compared ol and 04-mini. The detailed evaluation results and analysis of this issue are provided in Appendix J.4]
Ablation Study We report ablation studies on ol examining different prompt modality designs, separating grounded annotations into two layers, and adding in-contexi examples. Additional ablations that vary the object detection and OCR models are provided in Appendix 0.5
Prompt modality. Table 2 a) shows that using only visual prompts (Fig. Bal) or textual descrip-
tions (Fig. Ba)) results in a performance drop compared to combining both. This highlights their complementary roles in grounding infographic elements and supporting VLMs in chart understanding.
Prompt separation. Table bb) shows that separating the prompts into two layers leads to better performance than providing them in one layer (Fig. 3 B.). This suggests that reducing overlap through separation facilitates the visual grounding of infographic elements and improves chart understanding.

---

Published as a conference paper at ICLR 2026
CHINA HOW DOES
10
Whet por Ingresbie of
proportion of Chinese public respondents have neutral impression Japan?
Callorraa
FUEL
From the chart. Japan's 'very unfovorable is 38.4%, "somewhat untovorable' is 19.1%, the neutral portion labeled as 19.1%, so the neutral share 19.1%.
2125
Wrong!
HOWSD DOES CHHINA tcel
BL PIAARET
What proportion of Chinese public respondents have
neutral impression Japan?
Alphabet
First, Identify which row belongs to Japan by looking its label O Next, match the "Nautrral' catagory the legend neutrai-fase ican text appears belge/gray color the
noting the
numeric sogment given which reads '20.4%:
color
Correct!
Q shart bar sokkii
the culy and Elkasoracii
Figure 4: Grounded CoT guides the model to think step-by-ster and achieve the correct answer.
Table 2: Ablation of the grounded CoT method. The best one is bold.
(a) Prompt Modality
(b) Prompt Separation
(c) In-Conter Example
Visual
Textual
Textual
L-Layer
2-Layer
With Example
Without Example
6288
61.6
64.1
62.3
64.1
61.5
64.1
Incorporation of in-context examples. Table c)) shows that incorporating in in-context examples results in performance drop. This indicates that the latest VLMSs can perform reasoning tasks effectively without additional examples, which would instead introduce confusion and hinder performance.
4.2
EVALUATING OBJECT DETECTION MODELS
We evaluate the infographic element detection performance of 11 object detection models on InfoDet.
4.2.1
EXPERIMENTAL SETUP
Models Existing object detection models can be classified into two categories: foundation models that support zero-/few-shot detection and traditional deep learning models that require fine-tuning before detecting novel classes. We select the representative models in each category, including seven foundation models (RegionCLIP (Zhong et al.) 2022). Detic (Zhou et al. 2022). Grounding DINO (Liu et al. 2024), GLIP (i et al. 2022b), MQ-GLIP (Xu al. 2023). T-Rex2 (Jiang et al. 2024). and DINO-X (Ren et al.) 20244) and four traditional models (Faster R-CNN (Ren et al. (013
YOLOv3 (Redmon & Farhadi] 2018), RTMDet (Lyu et al. 2022). and Co-DETR (Zong et 2023
The above models are not tailored detecting charts and HROs. To address this, we evaluate three adaptation methods: 1) Zero-shot prompting, which uses text prompts to define target classes, 2) Few-shot prompting, which uses & randomly selected infographics to describe target classes, optionally augmented with text prompts, and 3) Standard fine-tuning, which updates model weights using annotated infographics, either with k random example infographics or the entire InfoDet training set. We compare the models at granularity levels: the element level targeting charts and HROs, and the mark level targeting sub-elements such as bars, axes, and legends. For both levels, the performance is measured using the average precision (AP) and recall
Evaluation protocol
7

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation results of the foundation and the traditional models at the element level. The best one is bold.
Zero-shot prompting
(b) Few-shot prompting, 4-shots
Average Precision (AP)
Average Recall (AR)
Average Precsion (AP)
Average Recall (AR)
Chart
HRO
Chart
HRO
Chart
HRO
Chart
HRO
RegionCLIP
MQ-GLIP TiRe22
Detic
188
23.7
11.3
122
16.2
21.8
24.7
Grounding DINO
12.6
12.2
63.2
46.0
GLIP
13.5
11.2
44.9
33.2
MO-GLIP
13.5
11.2
44.9
33.2
DINO-X
(c) Standard fine-tuning tonittuning...... 4-shots
(d) Standard fine-tuning InfoDet
Averuge Precsion (AP)
Averuge Recall (AR)
Average Precsion (AP)
Averuge Recall (AR)
Chari
HRO
Churi
HRO
Charl
IRR
Chart
HRO
Lig
RegionCILPP
Denic
19.6
14.2
37.0
22.8
Delic
39.6
34.3
57.4
47.7
Faster R-CNN
3.4
1.0
10.8
1.5
Faster R-CNN
78.9
49.0
80.8
52.7
YOLO.5
5.5
4.0
16.2
13.1
YOLO<3
14.7
25.5
43.2
35.7
RTMDei
RTMDet
59
Co-DETR
(AR) on the InfoDet test set. Please refer to Appendix 1.2 for more details on text prompts, fine-tuning hyperparameters, and computational costs.
4.2.2 RESULTS AND ANALYSIS
Comparing adaptation methods and object detection models at the element level We evaluate all applicable adaptation methods for each model, except for standard fine-tuning, which restricted to models that fit within the memory constraints of NVIDIA Tesla V100 GPU. For few-shot prompting and fine-tuning methods, we use k = 4, 10, and 30 randomly selected infographics. We average the results over 3 runs, excluding T-Rex2 and DINO-X X due to their reliance charged APIs. Table 3 shows the results for all models with = 4. The full results, including the variance across runs, available in Appendix K.1 We present key findings as follows:
Zero-shot and and few-shot prompting exhibit limited performance. Zero-shot prompting exhibits limited performance. As shown in Fig. [J]a,,
state-of-the- of--hh--rttttttt foundation models like DINO-X fail to interpret these concepts through textual prompts, often missing key components. Contrary to prior findings (Madan al. 2024), providing annotated example infographics does not lead notable improvements (Fig. ((6)). We attribute this to the models' pretraining on natural scenes (Mathew et al. 2022). which provides limited exposure to graphic representations such infographics. Consequently. the models lack the prior knowledge needed to effectively learn from the provided examples.
A TFO COUNTRIES
A'ALEE OF TWO COUNTRIES
A TALE OF TWC COUNTRIES
A TALE TWO COUNTRIES
(d)
Figure 5: Detection results of evaluated object detection models: zero-shot prompting with DINO-X: (b) 4-shot prompting with T-Rex2; (c) 4-shot fine-tuning with Co-DETR; (d) fine-tuning
on InfoDet with Co-DETR. Bounding boxes in colors are the predictions for charts and HROs
8

---

mAR Published as a conference paper at ICLR 2026
Table 4: Evaluation results of the traditional models the element level. The best one is bold.
Model
mAP
Faster R-CNN
47.0
49.9 1 0.1
YOLOv3
11.3 +.9
21.5
+ 0.5
RTMDet
46,9 0.2
59.5
+ 0.5
Co-DETR
69.6 0.4
77.0
0.3
Standard fine-tuning improves performance. Compared with zero-Ifew-shot prompting, fine-tuning with example infographics or the InfoDet training set yields higher performance. Few-shot exper- iments show that the performance improves significantly as the number of example infographics increases. Moreover, fine-tuning on InfoDet consistently outperforms few-shot fine-tuning, particu- larly for the traditional models. This is verified by Co-DETR's more accurate detection results after fine-tuning on InfoDet (Fig. d)) compared to using 4 example infographics (Fig. [Jc)).
Evaluating traditional models at the mark level Due to the effectiveness of fine-tuning traditional models at the element level, we further evaluate them the mark level. Table the performance of each model averaged over 3 runs. Despite the increased difficulty of mark-level detection, which leads to drop in mAP (e.g., from 76.1% to 69.6% for Co-DETR), the models still perform effectively, supported by the large-scale training set of InfoDet. We also evaluate model generalizability via cross-dataset evaluation following Deng et al. (2023). Specifically, we fine-tune each model on either our infographics or VG-DCU (Dou et al. 2024) (plain charts with mark-level annotations) and test on the other. Models show smaller drop in mAP when transferred from our infographics to VG-DCU than vice versa. For example. the mAP of Co-DETR drops 17.1% when transferring from our infographics to VG-DCU, compared with a much larger 53.7% 7% drop the opposite direction. This indicates that models trained on our infographics generalize more effectively due to the inclusion of infographic charts. The full experimental setup and results are provided in Appendix K.T
4.3 APPLYING THE DEVELOPED MODEL TO GRAPHIC LAYOUT DETECTION
To demonstrate the broader applicability of InfoDet, evaluate its effectiveness graphic layout detection tasks by applying the InternImage-ban based model.
4.3.1 EXPERIMENTAL SETUP
We evaluate the Internlmage- based model on three graphic layout detection datasets, Rico (Deka et al., 2017), DocGenome (Xia et al. 2024), and PosterLayout (Hsu et al.) 2023). As shown in Table we fine-tune five model variants for each dataset. Four of which are pre- traitrd on a different combination of Tma (Deng et al. 2009), Objects365 (Shao al., 2019), COCO (Lin et al. 2014), and InfoDet. The fifth variant is pre-trained on all four datasets and leverages a hierarchy- guided matching (HGM) scheme on InfoDet that encourages matched predictions preserve the hicrarchical relationships between charts and their sub-elements in the ground truth. Please refer to Appendix I.3 for more details on the graphic layout detection datasets, the hierarchy-guiiee matching scheme, fine-tuning hyperparameters, and computational costs.
4.3.2 RESULTS AND ANALYSIS
pre-training on InfoDet improves model performance when fine-tuned on these datasets, both on its own and when combined with existing large-scale pre-training data. With the growing interest in integrating multiple datasets for training foundation models (Yang et al. 2024).
As shown in Table 5
Table Performance of the detection models with different pre-training. The best one bold.
Pre-Training
Rico
DocGenome
PosterLayout
42.1
69.0
62.5
InfoDet
50.6
74,4
73.6
ImageNet-22# Objects365, COCO
51.8
78.7
74.9
ImageNet-222 Objects365, COCO, InfoDet
53.6
80.0
76.0
ImageNet-22K, Objects365, COCO.
InfoDet w. HGM
54.4
80.8
76.4

---

Published as a conference paper at ICLR 2026
InfoDet serves useful addition to existing resources for graphic layout detection. Additionally, incorporating HGM yields additional gains, suggesting that exploiting the hierarchical structure in InfoDet is a promising direction for future research.
CONCLUSION
In this paper, we introduce InfoDet, dataset designed to support infographic element detection. It features a diverse collection of real and synthetie infographics, along with bounding box annotations for texts, charts, HROs, and finer-grained sub-elements. Three applications demonstrate that InfoDet is not only valuable for developing visual reasoning methods but also broadly applicable to tasks such as object detection model evaluation and graphic layout detection. Although InfoDet has proven effective, there remain promising directions for future work. For example, analyzing the annotated infographics uncover design principles could advance automated infographic design.
ETHICS STATEMENT
To ensure the integrity of this work, we carefully consider several ethical aspects when collecting real infographics from online platforms. First, we manually collect the real infographics and release only their source URLs on Hugging Face, without hosting or redistributing third-party content. We have reviewed the data usage policies of the platforms and confirmed that they either explicitly permit (e.g., Statista, Visual Capitalist) or do not prohibit (e.g., Daily Infographics, Infographics Archive) the use of their content for research purposes, including the sharing of source URLs. Details of the data usage policies and licenses of each platform are provided in Appendix B Second, to exclude harmful and sensitive content from dataset, we: 1) collect only from reputable public platforms, which generally filter such content; and 2) use GPT-4o mini to flag potentially harmful or sensitive infographics, which then manually verified and removed. Finally, release dataset and model strictly for academic research purposes.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under grant U21A20469. The authors would like to thank Duan Li for his valuable contributions to the discus- sions.
REFERENCES
Leilani Battle, Peitong Duan, Zachery Miranda, Dana Mukusheva, Remco Chang, and Michael Stonebraker. Beagle: Automated extraction and interpretation of visualizations from the web. In Proceedings of the CHI conference on human factors in computing systems, pp. 1-8, 2018.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4291-4301, 2019.
Bokeh. Bokeh: Python library for interactive visualization, 2018. URL arq/en/latest/
Nadav Borenstein, Karolina Stanczak, Thea Rolskov, Natacha Klein Käfer, Natália da Silva Perez, and Isabelle Augenstein. Measuring intersectional biases in historical documents. In Findings of the Association for Computational Linguistics, pp. 2711-2730, 2023.
Michelle A Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva. Beyond memorability: Visualization recognition and recall. IEEE transactions visualization and computer graphics, 22(1):519-528 2016.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
10

---

Published as a conference paper at ICLR 2026
Yuwei Chen and Ming-Ching Chang. Transformer based image-text consistency analysis for info- graphic articles. In IEEE International Conference on Multimedia Information Processing and Retrieval, pp. 47-52, 2023.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the annual ACM symposium on user interface software and technology, pp. -45, 2017.
Dazhen Deng, Yihong Wu, Xinhuan Shu, Jiang Wu, Siwei Fu, Weiwei Cui, and Yingcai Wu. Visim- ages: A fine-grained expert-annotated visualization dataset. IEEE Transactions on Visualization and Computer Graphics, 29(7):3298-332 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and patlern recognition, pp. 248-255, 2009.
Shuguang Dou, Xinyang Jiang, Lu Liu, Lu Ying, Caihua Shan, Yifei Shen, Xuanyi Dong, Yun Wang, Dongsheng Li, and Cairong Zhao. Hierarchically recognizing vector graphies and A new chart- based vector graphics dataset. IEEE Transactions on Pullern Analysis and Machine Intelligence, 46(12):7556-7573 2024.
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. PP-OCR: A practical ultra lightweight OCR system, 2020. URL https: / arxiv.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2): 303-338, 2010.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Clarissa Hohenwalde and Antonios Hazim. Enhancing graphic accessibility for blind and visually impaired people: A systematic comparison of vision language models (vlm). INFORMATIK, pp. 613, 2025.
Enamul Hoque, Parsa Kavehzadeh, and Ahmed Masry. Chart question answering: State of the art and future directions. Computer Graphics Forum, 41(3):555- 5722, 2022.
Hsiao Yuan Hsu, Xiangteng He, Yuxin Peng, Hao Kong, and Qing Zhang. Posterlayout: A new benchmark and approach for content-aware visual-textual presentation layout. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6018-600 6026, 2023.
Kevin Hu, Snchalkumar's Neil'S S Gaikwad, Madelon Hulsebos, Michiel A Bakker, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, and Çağatay Demiralp. Viznet: Towards a lcrge visualization learning and benchmarking repository. In Proceedings of the CHI conference human factors in computing systems, pp. 1-12, 2019.
Huang, Hou Pong Chan, Yi Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering. 37()): 2550-2568, 2024.
IPTC Media Topics taxonomy. Iptc media topics taxonomy, 2025. URL httthttss//ppc..rrr standards/media topics/ Accessed: 2025-02-04.
JaidedAI. Easyoer: Ready-to-use ocr with 80+ EasyOCR 2024.
Tanuj Jain, Christopher Lennan, Zubin John, and Dat Tran. Imagededup. htmm:ideall//maggeee 2019.
https://gittae

---

Published as a conference paper at ICLR 2026
Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via sext-visual prompt synergy. In Proceedings of European Conference on Computer Vision, pp. 38-57, 2024.
Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kådár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yes Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015 4026, 2023.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language- image pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10965-10975, 2022b.
Ming Li, Ruiyi Zhang, Jian Chen, and Tianyi Zhou. TRIG-bench: A benchmark for text-rich image grounding. In Workshop on Reasoning and Planning for Large Language Models at ICLR, 2025a.
Zhen Li, Yukai Guo, Duan Li, Xinyuan Guo, Bowen Li, Lanxi Xiao, Shenyu Qiao, Jiashu Chen, Zijian Wu, Hui Zhang, Xinhuan Shu, and Shixia Liu. Chartgalaxy: dataset for infographic chart understanding and generation, 2025b. URL htthttps:: bbttts://aaxixy: http://axvvvv//// http:///waawaw.caaaaaa////////// 2505 18668
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145-151, 2002.
Minzhi Lin, Tianchi Xie, Mengehen Liu, Yilin Ye, Changjian Chen, and Shixia Liu. Infochartqa:
A benchmark for multimodal question answering on infographic charts, 2025. URL https
19028
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of European Conference on Computer Vision, pp. 740-755, 2014.
Shilong Liu, Tianhe Ren, Jiayu Chen, Zhaoyang Zeng, Hao Zhang, Feng Li, Hongyang Li, Jun Huang, Hang Su, Jun Zhu, et al. Detection transformer with stable matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6491-6500, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. Proceedings of European Conference on Computer Vision, pp. 38-55, 2024.
Shangbang Long, Xin He, and Cong Yao. Scene text detection and recognition: The deep learning era. International Journal of Computer Vision, 129(1):161-184, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. IconQA: new benchmark for abstract diagram understanding and visual language reasoning. In Proceedings of Neural Information Processing Systems Datasets and Benchmarks Truck, 2021.
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang. Yanyi Liu, Shilong Zhang, and Kai Chen. Rumdet: An empirical study of designing real-time object detectors, 2022. URL https://aran http://wxwiv.oooooo/////ooo/o///
owg. /ttp///aa22222/2. 07784
12

---

Published as a conference paper at ICLR 2026
Anish Madan, Neehar Peri, Shu Kong, and Deva Ramanan. Revisiting few-shot object detection with lasion-language models. In Proceedings of Advances in Neural Information Processing Systems, volume 37, pp. 19547-19560, 2024.
Dipu Manandhar, Dan Ruta, and John Collomosse. Learning structural similarity of user interface layouts using graph networks. In Proceedings of European conference computer vision, pp. 730-746, 2020.
Dipu Manandhar, Hailin Jin, and John Collomosse. Magic layouts: Structural prior for component detection in user interface designs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15809-15818, 2021.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics, pp. 2263-2279, 2022.
Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj. Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmo- hammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartqapro: A more diverse and challenging benchmark for chart question answering, 2025. URL https
7/arxiv. org /200////5504 05506
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga, In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv:1802.0344 2018.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. PlotQA: Reasoning scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020.
Microsoft. OCR Optical Character Recognition. perricesccomppteeriiiiioovvveeee
2024. Accessed: 2025-07-09.
o1. Learning to reason with llms. http:://wwaaiiooommeeeeeeemennnp 2025. Accessed: 2025-05-10.
o3. thinking-with-imagae 04-28.
http:://weenai.ccoo////c//hiiiie 2025. Accessed: 2025-
04-mini. 2025-04-28.
http:://ggaaaiiooommnmichhhhhiih 2025. Accessed:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of International conference on machine learning, pp. 8748-88 0783-776,,11 2021.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement, 2018 URL https
02767
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of Advances in Neural Information Processing Systems, volume 28, 2015.
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song. Yuhong Zhang, Hongjie Huang. Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, and Lei Zhang. Dino-x: A unified vision model for world object detection and understanding. 2024. URL https 7/ Tarxivv. wtt.vvv.orggbba////111/// 14347
13

---

Published as a conference paper at ICLR 2026
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9430-8439) 2019.
Statista. Statista, 2025. BtLhttps: /www. www.ssttttttcc..m com/
Accessed: 2025-02-04.
Visual Capitalist. Visual capitalist, 2025. URL hutpes//wwwviisualaaiaaaal
Accessed: 2025-02-04.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14408-14419, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouuht prompting elicits reasoning large language models. In Proceedings of Advances in neural information processing systems, volume 35, pp. 24824-24833 2022.
Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang. Chao Xu, Conghui He, Pinlong Cai, Min Dou. Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, and Yu Qiao. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models, 2024. URL https::ts
org/ abs/ 2406.
Yifan Xu, Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang, Ke Li, and Changsheng Xu. Multi-modal queried object detection in the wild. In Proceedings of Advances in Neural Information Processing Systems, volume 36, pp. 4452-4469, 2023.
Weikai Yang, Mengchen Liu, Zheng Wang, and Shixia Liu. Foundation models meet visualizations: Challenges and opportunities. Computational Visual Media, 10(3):399.424, 2024.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The International Conference Learning Representations, 2023.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-iman imaage pre- training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16793-16803, 2022.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In Proceedings of European conference on computer vision, pp. 350-360-88833 2022.
Jiangning Zhu, Zheng Wang, Zhiyang Shen, Lai Wei, Fengyuan Tian, Mengchen Liu, and Shixia Liu. Reorderbench: A benchmark for matrix reordering. IEEE Transactions on Visualization and Computer Graphics, pp. 1-15, 2025.
Chen Zhu-Tian, Yun Wang, Qianwen Wang, Yong Wang, and Huamin Qu Towards automated infographic design: Deep learning-based auto-extraction of extensible timeline. IEEE transactions on visualization and computer graphics, 26(1):917-926, 2020.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6748-6758. 2023.
14

---

Published as a conference paper at ICLR 2026
A STATISTICAL COMPARISON OF INFODET WITH EXISTING CHART DATASETS
Table 6 provides a statistical comparison of InfoDet with existing datasets. Unlike most existing datasets that focus plain charts, InfoDet is specifically designed for infographic charts, where charts, texts, and HROs are tightly integrated in visually complex layouts. Compared to the dataset by |Brrinn al. (2016), the only existing infographic dataset with mark-level annotations, InfoDet is significantly larger in scale and better suited for training object detection models.
Table 6: Statistics of existing chart datasets.
Dataset
Real
# Synthetic
Infographic?
InfoDet (ours)
11, 264 393
90, 000
Borkin et al. (016)
0
QA Kahou
2018
100, 000
PlotQA (Methani et al 2020)
0
224, 377
Beagle BBtttee et al. 2018)
41, 000
Vislmages (Deng et al. 2023
12, 267
0
VG-DCU (Dou et al. 2024)
4, 515
10, 682
B ONLINE PLATFORMS FOR REAL INFOGRAPHIC COLLECTION
We collect the real infographies from the 10 online platforms listed in Table 7 The infographic collection strictly adheres to the copyright and licensing regulations of the respective platforms.
Table 7: Infographic platforms and licenses.
Platform
Website Link
Licenses BCBNC
Statista
statista.com
Visual Capitalist
visualcapitalist.com
Customized
"For individuals and small organizations (5 people or less, or < $1 million in rev- enue), we allow you to use our visuals variety of use cases for free. These include personal and commercial use cases, such as: embedding our graphics in a newsletter, report, video, presenta- tion, or on your website."
World Statistics
world-statissiccs.og
CC
Our World Data
ourwolldndata.org oecd.org
CC
OECD
BY
Openverse
openverse.org
CC
The Conversation
theconversation.com
CC BY-ND
Kaiser Family Founda-
kff.org
CC BY-NC-ND
tion
Daily Infographics
dailyinfographics.com
Customized
No prohibition on research use or sharing of source URLs.
Infographics Archive
infographicsarchive.com
Customized
No prohibition research use or sharing source URLs.
C EXAMPLE SYNTHETIC INFOGRAPHICS
We employ a template-base based method to create synthetic infographics. Fig. 6 shows examples of design templates and infographics generated from them.
15

---

Published as a conference paper at ICLR 2026
6886 m
TEXT
Polymelams Eacond
Syrtheti Spen, More Chamican
Traveeed
Diebetes Prevelence
Ferer Corrih douara
Highest Mexico
TEXT
(a)
(b)
Figure 6: Template-based generation of synthetic infographics: (a) design templates; (b) synthetic
infographics generated from the design templates.
D DATASET STATISTICS
Fig- shows the distribution of the number of annotated texts, charts, and HROs per real and synthetic infographic. On average, each real infographic contains 53.76 text elements (each corresponding to a line of text), 1.94 charts, and 14.74 HROs, while each synthetic infographic contains 74.21 text elements, 3.20 charts, 10.16 HROs, and 61.69 sub-elements. The slight difference in annotation density between real and synthetic infographics enhances the diversity of the dataset, improving its utility for training models to handle diverse infographics.
I5ce
Norber Raa a0gg a/1
2000
4000
2000
2000
Humber
2000
2000
500
175
23400
Synthers Prggrppke
416881
10000
15000
25000
20000
20000
15,00
3000
10000
10000
3000
3.50
200
300
400
200
20
100
Sumber of Charts
Number of 190O
Figure 7: The distribution of the number of texts, charts, and HROs in each infographic.
We classify charts and HROs into subcategories: charts are categorized into 75 distinct types, while HROs are labeled as either data-related or theme-related objects. The 75 chart types are: 1) Vertical bar chart, 2) Vertical stacked bar chart, 3) Vertical grouped bar chart, 4) Horizontal bar chart, 5) Horizontal stacked bar chart, 6) Horizontal grouped bar chart, 7) Radial bar chart, 8) Radial stacked bar chart, 9) Radial grouped bar chart, 10) Circular bar chart, 11) Circular stacked bar chart, 12) Circular grouped bar chart, 13) Pictorial percentage bar chart, 14) Histogram, 15) Lollipop chart,
16

---

Published as a conference paper at ICLR 2026
16) Dot chart, 17) Diverging bar chart, 18) Vertical bar chart with circle, 19) Horizontal bar chart with circle, 20) Vertical dot bar chart, 21) Horizontal dot bar chart, 22) Dumbbell plot, 23) Span chart, 24) Bump chart, 25) Line graph, 26) Spline graph. 27) Stepped line graph, 28) Slope chart, 29) Small multiples of line graphs, 30) Small multiples of spline graphs, 31) Small multiples of stepped line graphs, 32) Area chart, 33) Spline area chart, 34) Layered area chart, 35) Layered spline area chart, 36) Range area chart, 37) Stacked area chart, 38) Radial area chart, 39) Radial spline area chart, 40) Radial layered area chart, 41) Radial layered spline area chart, 42) Radial range area chart, 43) Radial stacked area chart, 44) Diverging area chart, 45) Diverging spline area chart, 46) Small multiples of area charts, 47) Small multiples of spline area charts, 48) Pie chart, 49) Donut chart, 50) Semicircle pie chart, 51) Semicircle donut chart, 52) Rose chart, 53) Small multiples of pie charts, 54) Small multiples of donut charts, 55) Small multiples of semicircle pie charts, 56) Small multiples of semicircle donut charts, 57) Small multiples of rose charts, 58) Radar line chart, 59) Radar spline chart, 60) Small multiples of radar line charts, 61) Small multiples of radar spline charts, 62) Proportional area chart, 63) Scatterplot, 64) Grouped scatterplot, 65) Bubble chart, 66) Heatmap, 67) Waffle chart, 68) Small multiples of waffle charts, 69) Alluvial diagram. 70) Gauge chart, 71) Small multiples of gauge charts, 72) Funnel chart, 73) Pyramid chart, 74) Treemap, 75) Voronoi treemap.
For the real infographics, we have attempted classify the charts and HROs using GPT-4o. However, achieves limited accuracy, with 61.49% on 1, 179 charts and 74.69% on 1, 498 HROs. To com- plement the dataset with chart-type annotations, we have attempted two more methods: 1) GPT-5; and 2) a two-stage CLIP-based linear probing method that first predicts one of 10 coarse categories (e.g., bar, line) and then refines it into one of 75 chart types using classifiers trained on our synthetic infographics. GPT-5 achieves an accuracy of 69.64%, while the two-stage method achieves 78.800 80%. We manually verified the chart-type annotations produced by the two-stage method and released the verified annotations on Hugging Face.
E EXPERT BACKGROUND AND REFINEMENT ROUNDS
Expert background Seven experts participated in the annotation process. Two are Ph. D. students majoring in data visualization who are co-authors of this paper. The other five are experienced annotators from a vendor company, each with at least two years of experience in object detection annotation, practical experience in plain chart annotation, and at least a bachelor's degree.
Refinement rounds The annotation refinement process spanned four months, with one round lasting two weeks, resulting in total of eight rounds. Each round followed this procedure: 1) The Ph.D. experts reviewed existing annotations to identify potential issues; 2) They trained the remaining annotators to perform targeted corrections; 3) The detection model was fine-tuned on the corrected annotations to improve subsequent predictions. We qualitatively observed that the annotation quality, as well as the detection accuracy, steadily improved across the refinement rounds. Since evaluating annotation precision and recall requires additional human effort, we evaluated precision and recall at three stages: before refinement, after the fourth round, and after all rounds. shown in Table the precision and recall steadily improved. The refinement rounds required around $3,000 in human annotation cost and 313 GPU-hours for model fine-tuning on an NVIDIA Tesla V100, priced at approximately $0.7 per GPU-hour. In total, the refinement process cost roughly $3.220. In contrast, fully manual annotation of all real infographics is estimated to cost more than $30,000, due to the expertise required and the need for multi-rount quality verification to ensure annotation accuracy. It also produces a model that can generate reliable annotations for future chart-related datasets, thereby reducing annotation costs.
17

---

Published as a conference paper at ICLR 2026
Table 8: Annotation precision and recall across the refinement process.
Refinement Round
Precision
Recall
Before refinement
69.6%
77.0%
After the fourth refinement round
89.0%
91.3%
After all refinement rounds
93.9%
96.7%
To ensure high inter-annotator agreement in the refinement rounds, the five annotators trained each round by the two Ph.D. experts, and their corrections were required to reach an average IoU of 0.95 before proceeding annotation. To further verify the consistency of the Ph.D. annotators, we conducted an additional inter-annotator agreement study on 1, 1,000 real infographics. For each infographic, first generated annotations using the model before refinement, then randomly selected one element from those annotations. The Ph.D. experts independently corrected the element, adjusting its location and category as needed. They were also asked to mark the occurrences of ambiguous chart-HRO boundaries. Table shows the inter-annotale agreement between the two Ph.D. experts on all elements and on 39 elements with ambiguous chart-HRC HRO boundaries. The experts performed consistent corrections (average 1oU=0.95, category agreement=0.99) These corrections differed from the model predictions, which showed lower agreement (average loU=0.90 category agreement=0.96)) indicating that they can effectively mitigate systematic biases in the model annotations. Even for the elements with ambiguous boundaries, the agreement remains high (average loU=0.84, category agreement=0.92), further confirming the reliability of the corrections.
Average IoU
Category Agreement
Average IoU
Category Agreement
Expert-Expeut Expert-Model
0.95
0.99
Expert- Expert
0.84
0.92
0.90
0.96
Expert-Model
0.77
0.91
(a) Results on all elements
(b) Results on elements with ambiguous boundaries
Table 9: Inter-annotator agreement study results between two Ph.D. experts.
F LIMITATIONS
Dataset scope Regarding the images, InfoDet focuses on infographics that combine texts, charts, and HROs, while images lacking charts or HROs are excluded during collection. Consequently, InfoDet less suitable as the sole training source for applications targeting graphic designs without charts or scientific documents without HROs. Regarding the annotations, model-in-the-loop method utilized to generate annotations for the real infographics. This method helps maintain consistent annotation guidelines across the dataset, but it may also introduce consistent errors in ambiguous cases. This should taken into account for tasks requiring highly precise detection results.
Remaining model errors At the end of the refinement process, we randomly sample 1, 250 infographics to evaluate the quality the generated annotations. While the precision (93.9%) and recall (96.7%) are comparable to those of widely used object detection datasets, small number of annotation errors still remain. We have summarized these errors in Table 10 The most frequent errors include: 1) imprecise bounding box localization due to ambiguous element boundaries; 2) marks or annotations mistakenly detected as HROs due to their high visual similarity; and 3) missed tiny elements due to their limited scale.
18

---

Published as a conference paper at ICLR 2026
Table 10: Distribution of remaining annotation errors after the refinement process.
Error Type
Percentage
Imprecise bounding-bol box localization
36.5%
Marks or annotations detected HROs
32.9%
Missed tiny elements
20.4%
Others
10.2%
G CREATING MARK-LEVEL ANNOTATIONS
To create mark-level annotations, extend our synthetic infographic creation and an- notation methods. Using these methods, we generate annotations of 26 element cate- gories. The 26 categories are: "vertical_griidinn end", "bar_mark' "proportional-area__ marr",,ar"", "axis", Nother_gridling line_mark",
"dumbbell mark", "scatter paaterr___rr,,,_ "leg-
Viine_mrk"" "area-under- "bump_mare "horizontal griizonnalggiillinee, "stacked-bar ball-bbbrrrrr,,,,.-- "radar_mark" "donut mark". fonkky__mrkee "pie_marke mark", "span mark", "bubble mark", "histogram mark", "treemap mark", "waffle mark", "pyramid mark", "funnel mark", and "range mark".
H DATASET ANALYSIS
We have conducted analyses verify that our dataset exhibits high diversity, no harmful bias, and high fidelity of synthetic infographics.
H.1 DIVERSITY ANALYSIS
To evaluate the diversity the dataset, we extract set of attributes from each infographic and examine the variety of values for each attribute. Specifically, consider four key attributes: chart type, infographic topic, visual style, and communication tone. Attribute values are extracted using Gemini-2.5-llash, with slightly different strategies for each attribute: Chart type is selected from the 75 identified chart categories; 2) Infographic topic was categorized based on the [PTC Media Topics laxonomy, a widely adopted taxonomy for news content; tone, the model first generate a descriptive term for each infographic, and then grouped semantically similar terms into attribute values. We report the number of unique values identified for each attribute in Table [T] These numbers indicate broad range of chart types, infographic topics, visual styles, and communication tones. In particular, the infographic topics span all 17 top-level categories and 96 out of 120 second-level topics in the IPTC Media Topics taxonomy. These results verify the diversity of our dataset.
For visual style and communication
Table 11: Diversity of infographic attribute values in InfoDet.
Attribute
# Attribule Values
Example Attribute Values
Chart Type
75
horizontal bar chart, line graph, treemap public health, weather forecast, market and exchange minimalistic, cartoonish, vintage neutral, critical, persuasive
Infographic Topic
96
Visual Style
15
Communication Tone
21
H.2 BIAS ANALYSIS
To mitigate potential bias, we have monitored attribute distributions during the infographic collection and generation and applied resampling techniques when necessary. For example, we observed an overrepresentation of bar charts in early synthetic samples and reduced their frequency through targeted resampling. We also computed pointwise mutual information (PMI) (Borenstein et al. 2023) across attribute pairs identify unexpected co-occurrences. We have manually inspected all pairs with PMIs greater than and found that the high-PMI pairs (e.g., environment topic + concerned tone) aligned with common communication patterns and did suggest harmful or misleading bias.
19

---

Published as a conference paper at ICLR 2026
In summary, found no harmful or systemic bias in the dataset and ensured that attribute variations remain reflective of real-world usage.
H.3 QUALITY EVALUATION OF SYNTHETIC INFOGRAPHICS AND COMPARISON WITH REAL INFOGRAPHICS
Quality evaluation The quality of synthetic infographics evaluated at both the sample and dataset levels. At the sample level, we randomly select 500 synthetic infographics and manually verify their quality. All of them clearly convey the underlying data, contain HROs that align well with their intended semantics, and have accurate annotations. A minor issue is observed in 18 samples, where slight unintended overlap occurs between elements due to minor misalignments during rendering. However, this does not affect object detection, the boundaries of the overlapping elements remain clear. At the dataset level, we evaluate how well the synthetic infographics cover the real ones in feature space. Specifically,
CLIP (Radford et al. 2021) extract the feature embeddings of all infographics and project them into a two-dimensional space using UMAP (MeInnes et al., (2018). 8). The space is divided into uniform grid, and real infographic is considered "covered" at least one synthetic infographic falls in the same grid cell. The results show that 92.64% of the real infographics are covered, indicating the high representativeness of the synthetic infographics.
Comparison with real infographies We qualitatively examine the difference in the distribution of synthetic and real infographics in the UMAP visualization (Fig. 8). While the overall distribution of synthetic and real infographics largely overlaps, some distinctive characteristics Synthetic infographics include chart variations specially crafted by design experts, such as those with -and-drawn-style fills, which relatively rare in real infographics. Real infographics, the other hand, uniquely feature composite charts, where multiple single charts are combined by sharing axes or overlaying each other.
observed.
Couerage lestrap Cooer- 66% byrlhikkuuel
Only by -dhetc
32-
Figure 8: The distribution of synthetic and real infographics.
Additionally, we have compared the distribution real and synthetic infographics across the attributes using Jensen-Shannon divergence (Lin 2002), commonly used measure of similarity between probability distributions. Table 12 shows the evaluation results. The divergences for infographic topics, visual styles, and communication tones low, suggesting strong alignment. The divergence for chart types is relatively higher because the synthetic infographics cover all chart types and include higher proportion of rare (e.g., circular bar charts, waffle charts) to ensure sufficient coverage.
20

---

Published as a conference paper at ICLR 2026
Table 12: Computational costs (GPU hours) for fine-tuning models on different datasets.
Attribute
Distance
Chart Type
0.149
Infographic Topic
0,053 Visual Style
0.022
Communication Tone
0.038
DETAILED EXPERIMENTAL SETUP
1.1 THINKING-WITH-BOKER VIA GROUNDED CHAIN-OF-THOUGHT
Prompts for the grounded chain-of-thought method and the baselines In the grounded chain-of- thought method, we prepend the grounded infographic elements to the creettoo-caatgooy-speeeee prompt used ChartQAPro (Masry et al. 2025). Below is an example input to the vision-language model.
Example Prompt for Grounded Chain-of- Slain-of-Thought
You will be provided with two versions of the sume infographic chart, each with certain elements highlighted. You will also be provided with the information lists of elements highlighted in the images. Each entry in the lists of elements follows the formal (ID, Content), where:
ID means the id the element.
Content means the content of the element.
HOWDDOES CHINA FUEL
What your pressice
This above image highlights non-texi elements enclosed in boxes, each labeled with a unique ID. Here is the list of elements:
(ID=A, Content="hunin recognizable object")
(ID=R R. Content=" "ontent="char"""
21

---

Published as a conference paper at ICLR 2026
CHINA HOW DOES
FUEL
JULL
This above image highlights text elements enclosed in boxes, each labeled with unique ID.
Here is the list of elements:
*****e* 000ee0 0 400000 8800908665
(ID=X, Content="text: HOW DOES
(ID=Y, Content="text QUESTION")
labeled elements intended support you your upcoming Please to and make them as needed during your thinking and analysis, and be sure to mention their IDs when doing so. For example:
1. Based the content in box ID I, (your finding about the box), or:
Based the relationships of box
N. (your finding based the boxcs).
Below is the image of original infographic chart, followed by your task:
HOWS CHINA
OUESTION
these aftii reghtes!
FUEL?
1146
You given factoid question that need answer based on the provided image.
You need think stop-by-step. your final should a single word, number, or phrase. If the question unanswerable based on the information provided image, your answer should unanswerable, not generate units. But if numerical units such as million, m, billion, B, or K are required, use the exact notation shown the chart.
If there multiple final answers, put them in brackets using this format [' Answerl', 'Answer2']. Remember think step-by-step and mention the IDs of the elements you used, and reply the following ISON format:
22

---

Published as a conference paper at ICLR 2026
"Steps": "The slep-by-slep thinking prucess with IDs mentioned.", "A": "Your answer."
What proportion of Chinese public respondents have neutral impression of Japan?
Question:
For the baselines, we use the same prompt as ChartQAPro. Below are examples of the input for the three baselines: direct prompting, -ottoh,tt and program-of-thouga
Example Prompt for Direct Prompting
HOWDOES CHINA FUEL
DUCSTION Thii COLITIN rigfoca!
Pm
( Goveeee i mome Qrenennie
You given factoid question that you need Your answer should be single word, number, or phruse. If the question unanswerable based on the information in the provided image, your
based provided image.
should be unanswerable. Do generate units. But if numerical units such million, m, billion, B, or K are required, use the exact notation shown in the chart.
there multiple final answers, put them brackets using this format ['Answrrl', 'Answer2"]. Remember generate the final answer only without any additional text! Question: proportion
mese public respondents have neutral impressi
of Japan?
23

---

Published as a conference paper at ICLR 2026
Example Prompt for Chain of Thought
HOW CHINA FUOL?
OOCSTION
These rightns!
174
dittboe
You are given a factoid question that you need to answer based on the provided image.
to think step-by-slep, your answer should a single word, numbeer, or phrase. If question is unanswerable based on the information in the provided image, your answer should be unanswerable. Do not million, m, billion,
generate But numerical units the chart.
are required,
notation shown
there are multiple final answers, put them in brackets using this format I A'nswerl', 'Answer2'1 ].
Remember think step-by-step and format the final answer in separate sentence like "The answer is X" Question: What proport ian of Chi nese publ ie respondent have a neutral impression of Japan?
Example Prompt for Program-of of- -foooof-ooooe Thought
HOWDDOES DOES CHINA FUE22
Whst your prrsicce
Te
(i is -oee Qo
You given factoid question that you You need to write an executahle python code that calculates and prints the final answer, but your final answer should a single word, number, or phruse. If the question is unanswerable bused on information in the provided image, your answer should be unanswerable. Do not generate units. But if numerical units such as million, billion, B, are required, use the exact notation shown in the chart.
based on provided image.
If there multiple final answers, put them in brackets using this format A'Anwwrl', 'Answer2']
Remember return python code only without any additional text.
24

---

Published as a conference paper at ICLR 2026
Question: What progortion of Chinese public respondents have neutral impression of Japan?
Comparison of the visual prompts rendered in layer versus two layers In our grounded Chain-of-thoughe method, we propose to separate the visual prompts into two layers; one for charts and HROs, and the other for texts. As shown in Fig. 9 this separation improves visual clarity by reducing overlap between bounding boxes.
-K POL
RUS $2.2T
POL
ITA
RUS $2.2T
POL
ITA O
RUS S2.2T RR
ITA $2.51
SOL38
S2.5T F5
SPI38
$2.5T
SPI3B
E
FYLK
-0 ESP
HUN
HUN
+
AIFCPJ
AM
ESP
FFF
ESP
ROU 311
FD
$1.8
RT
$1.8T
REU D HH
rO
$1.81
W
DRK
FX
SWY
+
DNK
SWE +
FV
DNK
CI
N8D
NO
NOR 4
PRT
6ML
VISUAL CAPITALIST
VISUAL CAPITALIST
VISUAL CAPITALIST
X2
Figure 9: Comparison of the visual prompts rendered with different layer configurations: (a) visual prompts rendered in one merged layer: (b) visual prompts rendered in two separate layers.
Enhanced relaxed accuracy implementation Following CharlQAPro, we use the enhanced relaxed accuracy evaluate the chart understanding performance. This metric evaluates answers based on the following criteria:
1. Numeric answers are allowed 5% error margin.
2. For answers in 'years', exact match is required.
3. Textual answers are evaluated using the ANLS score (Biten et al. 2019). which is based on the edit distance between texts.
4. Multiple-choice and fact-checking tasks are evaluated using an exact-match criterion.
To more accurately evaluate model performance, make three refinements to the official implemen- tation of the enhanced relaxed accuracy:
1. We remove punctuation marks (i.e., commas and periods) from answers, ensuring that '25,000' and '25000' are treated as equivalent.
2. We remove unit symbols when evaluating numeric answers, that values like '100' and '$ 100' are treated as equivalent.
3. We standardize ratios and percentages by converting them into decimal form, so that expressions like '3:2', '150%', and '1.5' are all treated as equivalent.
I.2 COMPARING OBJECT DETECTION MODELS
We evaluate the performance of existing object detection models in detecting charts and HROs. As the models are not tailored detecting charts and HROs, adapt them using three adaptation methods: 1) Zero-shot prompting, which uses text prompts to define target classes, 2) Few-shot prompting, which uses & randomly selected infographics to describe target classes, optionally augmented with text prompts, and 3) Standard fine-tuning, which updates model weights using annotated infographics, either with ls random example infographics the InfoDet training set.
For zero-shot prompting, we evaluate six models: RegionCLIP (Zhong et al. 2022). Detic (Zhou al. 2022), Grounding DINO (Liu al. 2024), GLIP (Li al. 2022b), MQ-GLLIP (Xu et al. 2023
25

---

Published as a conference paper at ICLR 2026
and DINO-X (Ren al. 2024). all of which take the class names "chart" and "human recognizable object" as input.
For few-shot prompting, evaluate two models: T-Rex2 (Jiang et al. 2024) and MQ-GLIP (Xu et al., 2023). For T-Rex2, we provide randomly selected infographics with bounding box annotations. For MQ-GLIP, provide the class names along with the selected infographics.
For traditional fine-tuning, we evaluate six models: RegionCLIP, Detic, Faster R-CNN Renn et al. 2013), YOLOv3 (Redmon & Farhadi 2018). RTMDet (Lyu al.] 2022). and Co-DETR Zong et al. 2023). For fine-tuning on the entire InfoDet training set, we train for epochs with batch size of B and learning rate of Ir. Table 13 shows the fine-tuning hyperparameters, which adhere the official settings, as well as the computational costs, in terms of GPU hours using NVIDIA GeForce RTX 4090 D. For few-shot fine-tuning we adjust the number of training epochs inversely with the number of random infographics, ensuring consistent computational costs. All other fine-tuning hyperparameters remain unchanged.
Table 13: Training hyperparameters and computational costs for traditional fine-tuning on the entire InfoDet training set.
Hyperparameters
RegionCLIP
Detie
Faster R-CNN
YOLOv3
RTMDel
Co-DETR
Optimizer
SGD
AdamW
SGD
SGD
AdamW
AdamW
E
8
10
5
3
B Ir
1
8
64
64
64
64
3.75e
2e
1e
4e 3
le
Computational costs
20
40
(GPU hours)
We have compared the computational costs of fine-tuning the detection models InfoDet and existing standard datasets, including PASCAL VOC (Everingham et al. 2010)), COCO (Lin et al. 2014), LVIS (Gupta et al. 2019), and Objects365 (Shao al. (019)) in Table [4] The computational cost of fine-tund tunittnnnggann on Info is higher than that of Pascal VOC, comparable to that of COCO, and lower than that of Objects365 and LVIS.
Table 14: Computational costs (GPU hours) for fine-tuning models on different datasets.
Dataset
RegionCLIP
Detic
Faster R-CNN
YOLOv3
RTMDet
Co-DETR
PASCAL VOC
3
5
6
COCO
23
47
23
35
47
82
I.VIS
32
65
32
49
65
113
Ohjects365
119
238
119
179
238
416
InfoDet
20
40
20
30
40
70
I.3 APPLYING THE DEVELOPED MODEL TO GRAPHIC LAYOUT DETECTION
We evaluate the Internlmage-based model on three graphic layout detection datasets, Rico (Deka et al.) 2017) and DocGenome (Xia et al. 2024), and PosterLayout (Hsu et al. 2023). Rico contains over 66K user interfaces collected from Android applications. Following the common practice (Manandhar ct al. 2020 2021). we aim to detect 25 UI component classes and split the dataset into 53K 53K layouts for training and 13K for testing. DocGenome is large-scale scientific document dataset of 6.8M pages sourced from the arxiv repository, annotated with bounding boxes for 13 categories of components. We randomly sample 113K pages for training and 13K for testing. PosterLayout contains 9,974 posters annotated with bounding boxes of text, logo, and underly. We randomly split the dataset into 8,974 for training and 1,000 for testing. We fine-tune five model variants for each dataset. Four of which pre-tre tre-trained on different combination of ImageNet-22K (Deng et al.) 2009). Objects365 (Shao et al.) 2219), COCO (Lin et al.] 2014), and InfoDet. The fifth variant is pre-trained all four datasets and leverages hierarchy-guidee matching (HGM) scheme InfoDet that
26

---

Published as a conference paper at ICLR 2026
encourages matched predictions preserve the hierarchical relationships between charts and their sub-elements in the ground truth. Specifically, in the DINO detector, Hungarian matching determines one-to-one correspondences between predictions and ground-truti boxes for training loss computation. Prior works such as DN-DETR (Li et al.) 2022a) and Stable-DINO (Liu et al. (023) have shown that reliable matching process is crucial for stable model optimization. Motivated by this, incorporate hierarchy constraints, where chart sub-elements should be spatially contained within charts, into the matching process make more reliable. However, directly incorporating such hierarchy constraints turns the matching into quadratic assignment problem, which is NP-hard and intractable to solve in practice. Therefore, we adopt a two-stage matching method, first performing standard Hungarian matching and then refining the matching iteratively to satisfy the hierarchy constraints. Following the official setting (Wang et al. 2023), we fine-tune the frozen InternImage backbones along with the DINO detector (Zhang et al. 2023 for 12 epochs. The batch size is set to 16, and we use A optimizer (Loshchilov & Hutter 2019) with an initial learning rate of 0.0001 and a weight decay of 0.05. We use step- basp-baeed learning rate scheduler, which decreases the learning rate by factor of 0. at epochs 8 and 11. The training takes 196 GPU hours on Rico, 296 GPU hours on DocGenome, and 12 GPU hours on PosterLayout using NVIDIA Tesla V100.
J ADDITIONAL RESULTS AND ANALYSIS OF THINKING-WTT--OOIIS
J.1 RESULTS ON CHARTQA AND PLOTQA
To the generalizability of our method beyond ChartQAPro. we compare it against direct prompting. the best rest-performing baseline, on ChartQA (Hoque et al., 2022) and PlotQA (Methani et al., (2220) using ol. T5] shows the evaluation results. Our method performs better than direct prompting on both ChariQA and PlotQA, indicating that its benefits extend beyond ChartQAPro.
Method
ChariQA
PlotQA
Direct
75.6
30.6
Grounded CoT (ours)
79.2
31.3
Table 15: Performance of ol on ChartQA and PlotQA.
J.2 ADDITIONAL STATISTICAL ANALYSIS
To assess the robustness of the improvement, we conducted a repeated set of runs on o3, the model with the smallest improvement. As shown in Table in both runs, our method achieves the highest accuracy among all prompting methods. Even when accounting for variance, its mean performance remains clearly above the baselines.
Direct
CoT
PoT
Grounded CoT' (ours)
The first run
60.6
60.0
59.5
61.6
The second run
61.4
60.1
61.0
62.3
Meun Std
61.0 0.5
60.0 +:0::00
60.3 1.0
62.0 0.5
Table 16: Performance of o3 across two runs. The best score in each is bold.
27

---

Published as a conference paper at ICLR 2026
J.3 ADDITIONAL QUALITATIVE ANALYSIS
Fig. 10 shows how the provided grounded annotations elicit the model's reasoning capabilities. In Fig- To (a), direct prompting produces the incorrect answer "Switzerland" because it simply reads the second label. In contrast, grounded Chain-of Thouuhh first identifies the second-shores bar and then assigns it the correct label "Japan". Similarly, in Fig. (b). direct prompting misinterprets the relative bar heights between 2008 and 2013, while grounded Chain-of- Thouuhh correctly compares the visual differences by performing multi-step reasoning over them. These examples suggest that grounded Shain-of-Thought leverages the model's 's visual reasoning capabilities, mitigating errors that arise from relying on superficial visual cues rather than depending solely on prompt design.
COUNTRIES WITH HIGHEST GOLD RESERVES
courty Fay the bensl
COUNTRIES WITH HIGHEST GOLD RESERVES
eeatry tes the howril
the
Hroe this dia's, los Sactoociee bat 1.400 an
Firt, ideatiy shomae ticchat
Socent (id the lahel lhewe
Killow
The label Japas with the lower goil recanes.
Correct
TB tdbortap
(a)
|Piliion TRLI
LUPPRRAY VALUR IN IHE SOOCK EXCEANGL
compang male 2008 or 23137
PMilian F41.
CUMPANY VOEUE 1181 SIDER EXPFORULE
watch Hightr
company
vilis 3838 20137
funn cham, in tou 2013
2024 ligher company
2012
ualer rue that 200A
wake witre stack meltriis 2013
(b)
Figure 10: Additional qualitative comparison between direct prompting and grounded CoT.
J.4 DETAILED ANALYSIS OF ERRORS BY 03 ON CHARTQAPRO
Despite its strong visual reasoning capability, o3 achieves slightly lower accuracy compared to ol and o4-mini on the ChartQAPro benchmark (Masry et al., 2025). To investigate this, we randomly sample 200 question-answer pairs and analyze the failure patterns when using grounded CO'.. We identify two primary sources of failures: 1) perception error, where models fail to correctly interpret the content and relationships of the infographic elements, and 2) instruction following error, where models do not adhere to the prompt when formatting the answer. As shown in Table 17 perception errors are the main cause of chart understanding failures, occurring with similar frequency across all models. However, o3 shows a higher frequency of instruction-follon following errors, contributing to its slightly lower overall performance compared ol and o4-mini. In particular, even when instructed to output the numerical answer single word, 03 often includes extra words like 'xx' and 'about'. To address this, we have attempted to increase the reasoning effort from 'medium' to 'high'. However, shown in Table I8, this change does not yield an obvious improvement in the chart understanding performance, and the instruction following error still occurs with a similar frequency. This suggests that the 'medium' setting already provides sufficient reasoning budget for ChartQAPro. and alternative strategies are needed to enhance o3's s cnarucctio-followwiii ability.
Table 17: Error analysis of chart understanding failures on ChartQAPro for ol, o3, and 44-mini
Instruction Following Error
Model
Perception Error
ol
48
12
o3
47
22
04-mini
46
28

---

Published as a conference paper at ICLR 2026
Table 18: Performance of o3 using different levels of reasoning effort.
Reasoning Effort
Direct
CoT
PoT
Grounded CoT
Medium
60.6
60.0
59.5
61.6
High
60.4
61.0
60.8
61.8
J.5 ADDITIONAL ABLATION STUDIES
Varying detection model We replace our InternImage-baeed detector with two alternatives: Faster R-CNN, which performs worse COCO, and Co-DETR, which performs comparably COCO. As shown Table 19) the accuracy on ChartQAPro drops when using Faster R-CNN, while remains comparable when using Co-DETR, indicating that Grounded CoT benefits from stronger detectors and remains stable when the underlying detection quality is high.
Varying OCR model We replace our PP-OCRy4 with two alternatives: FasyOCR (JaidedAl 2024). commonly used baseline OCR model, and AzureOCR (Microsofi] (2024), one of the strongest commercial OCR systems. As shown in Table (9),, the accuracy on ChartQAPro drops when using EasyOCR, while it remains comparable when using AzureOCR, indicating that Grounded CoT can benefit from stronger OCR to some extent, but PP-OCRv4 is already sufficiently strong for this task.
Detection model
ChartQAPro
OCR model
ChartQAPro
Faster R-CNN (weaker)
63.4
EasyOCR (weaker)
62.9
Co-DETR (comparable)
63.8
Ours
64.1
Ours
64.1
AzureOCR (stronger)
64.0
(a) Detection model
(b) OCR model
Table 19; Additional ablation studies ol.
K DETAILED EVALUATION RESULTS AND ANALYSIS
K.1 DETAILED EVLUATION RESULTS
Comparing adaptation methods and object detection models We evaluate all applicable adapta- tion methods for each model, except for standard fine-tuning, which is restricted models that fit within the memory constraints of an Nvidia Tesla V100 GPU. For few-shot prompting and fine-tuning methods, use & 4, 10, and 30 randomly selected infographics. We average the results over 3 runs, excluding T-Rex2 and DINO-X, due to their reliance on charged APIs. Tables 20 and show the AP and AR along with their standard deviation for all models.
29

---

Published as a conference paper at ICLR 2026
Table 20: AP of object detection models for the chart and HRO categories. The best one bold.
Model
Zemoshot prompting
Few-shot prompting
10-shots
Standard fine-tuning
Chart Category
RegionCLIPP
0.83
681 3,70
9.38 + 1.62
10.12 = 0.96
10.14 + 0.80
Foundatin
Melcls
Grounding
Detic
1.77
19.62 =
23.02 1.38
28.00 1.42
39.57 4
GLIP
13.52
MQ-GLIP
13.52
16.24
0.63
16,80
0,19
16.91 0.24
T-Rex2
12.22
DINO-X
13.99
Tradiional Models
R-CNN
YOLOv3
5.49
1.64
5.121.2.88
7.83
14.70 5.92
RTMDel
12.82
1.74
25.90
2.44
28.84
2.09
83.65 + 3.46
Os-EETR
27.63 11.41
31,65
4.20
43.36
2.99
88.22 + 1.64
HRO Category
Region(11
0.23
1.38
0:11
Founbatio Models
Denic
4.38
14.24 5.15
22,47 + 3.29
30.41 = 1.19
34.31 + 0.59
Grounding Dino
12:24
GLIP
11:21
MQ-GLIP
11:21
15.46
16.18
16.94 0.29
DINO-X
14.94
Tradiional Mockls
Faster CNN
0.98 0.13
4,07 1: 1.05
11.59 = 1.01
49.02 0.17
YOLOv3
3.96 0.77
3.95:1..5 1,52
1.28
25.52 2.48
RTMDer
18.91
1.34
26.38
3.29
53.64 0.22
3694
362
Table 21: AR of object detection models for the chart and HRO categories. The best one is bold.
Model
Zero-shot prompting
4-ohoos
Few-shot prompting
10-shots
30-shots
4-00oos
10-shots
Standard fine-tuning
30-chots
InfoDer
Chart Category
Region(1
1.47
0.8
0.65
Founbston
Deric
23.72
36,99 + 6.02
40,40 + 0.97
44.06 = 1.39
57.38 D41
Mokel
Grounding Dino
63:12
GLIP
44.89
21.84
DINO-X
29.36
Tradtional Modkls
Faster C-CNN
10.79 =: 4.69
15.86 4: 2.74
20.86 = 4.79
80.84 4: 0.31
YOLOv3
16.21 2.54
15:98
5,08
23.33 - 1.55
43.16 1 5.45
HRO Category
RegionCLIP
24.92
22.85 1.55
26.21 4: 0.65
27 08.08
0.43
28.56 J: 0.27
Foundatin Modes
22,80
5.80
1.44
47.74
0.50
Grounding
GLIP
33.21
MQ-GLUP
33.20
40.72
2.06
41.69
1.45
42.38 0.54
T-Rex2
24.72
DINO-X
29.14
Traditonal Models
Faster
YOLOv3
13.06 1.76
18:75 :: 1.03
25.31
0.96
35.70 2.01
RTMDer
49. 0.69
49.72 0.94
52.50
1.15
59.94 0.14
Co-DETR
49.74 0.35
57.15
0.38
61.48 = 0,84
69.47 + 0.92
Cross-dataset evaluation between our dataset and VG-DCU We evaluate the generalizability of the traditional models via cross-dataset evaluation following Deng et al. al. (202)). Specifically, we train each model on either our infographics or VG-DCU (Dou et al. 2024). which comprises plain charts with element-level annotations, and evaluate each model on the other dataset. To support the transfer between datasets, we identify common categories with identical annotation guidelines, resulting in four categories: "bar mark". "line paceee_arr,_____ "pie pate__mar",, and "axis". We train and evaluate the models these categories. Table 22] shows the evaluation results. Models show a smaller drop in mAP when transferred from our infographics to VG-DCU compared to the opposite direction. This shows that models trained our data exhibit stronger generalizability due the inclusion of infographic charts.
30

---

Published as a conference paper at ICLR 2026
Table 22: Cross-dataset evaluation results
Training
VG-DCU
Ours VG-DCU
Test set
VG-DCU
Ours
mAP
Ours
mAP
Faster R-CNN
59.5
19.2
40.3
53.7
19.1
34.6
YOLOv3
26.4
4.1
22.3
23.2
12.5
10.7
RTMDet
70.2
28.1
42.1
60.9
42.4
18.5
Co-DETR
86.7
33.0
53.7
71.0
53.9
17.1
K.2 ADDITIONAL ANALYSIS
Post-fine-tuning regression on natural images Our object detection model is built by fine-tuning Internlmage-L with the DINO detector. After fine-tuning. the model no longer predicts natural- image classes, making direct evaluation on COCO or other natural- Developing unified model that robustly handles both infographics and natural images would indeed be highly beneficial, but it is beyond the scope of our current work, which focuses exclusively on understanding infographics.
benchmarks infeasible.
Mark-level performance drop The drop in detection performance from element-level to mark- level primarily due to two factors: 1) The mark-level annotations contain more classes with high visual similarity (e.g., "gauge_mare mark" and "donut moanut" which increases classification difficulty; and 2) The mark-level annotations contain smaller chart sub-elements, which makes localization harder. Boundary ambiguity is not critical factor because most mark-level categories correspond to clear geometric shapes with well-defined boundaries.
Why HRO detection is harder Analysis of the remaining errors of our Internlmage-based based model in Appendix Fereelals that the most frequent errors include: 1) imprecise bounding- localization due to ambiguous element boundaries (36.5%); 2) marks or annotations mistakenly detected HROs due to their high visual similarity (32.9%); and 3) missed tiny elements due to their limited scale (20.4%). The first and third errors occur more often for HROs, while the second error occurs only for HROs, making them harder to detect.
Which HRO type is more challenging Data-related HROs are more challenging detect for three reasons: 1) They are more likely to be closely integrated with charts, resulting in ambiguous boundaries; 2) Marks and annotations are more likely be mistakenly detected as data-related HROs; and 3) They are usually smaller and harder to localize.
Root cause of foundation model failure While foundation models achieve limited performance (mAP < 20), traditional models fine-tuned on our dataset can achieve much higher performance (mAP > 75) despite weaker inherent capabilities. This suggests that the primary cause of foundation model failure is pre-training data mismatch, rather than the intrinsic difficulty of the task.
Rico/DoeGenome components most improved by InfoDet pre-training Table 23 shows the AP changes of Rico and DocGenome component types when fine-tuned from scratch versus pre-trained on InfoDet. The top three component types benefiting most from InfoDet pre-training are Date Picker (+29.5), Slider (+17.4), and Radio Button (+16.6) in Rico, and Text (+11.0), Code (+8.5), and Title (+7.6) in DocGenome. We observe that the improvement on Rico 142.165 A=8.5) is larger than that DocGenome (69.0-744. A=5.4). This is because Rico's data contains complex UI
31

---

Published as a conference paper at ICLR 2026
hierarchies, which align well with InfoDet's 's chart and sub-element hierarchies and can thus better leverage the pre- training. whereas DocGenome components are largely flat.
Category
AAP
Date Picker
29.5
Slider
17.4
Radio Button
16.6
Card
13.5
Number Stepper
Category
AAP
Advertisement
12.8
Text
11.0
Checkbox
10.5
Code
8.5
On/Off Switch
8.9
Title
7.6
Input
Background Image
7.8
Caption
6,5
List Item
6.9
Footnote
Text
6.8
Equation
5.9
Text Button
Text-EQ
5.7
Map View
6.5
Figure
4.4
Icon
5.8
Algorithm
4.1
Modal
5.8
List
4.1
Image
Abstract
2.9
Web View
5.2
Table
Multi-Tab
4.7
Pager Indicator
4.7
PaperTitle
-2.5
Bottom Navigation
(b) DocGenome
Video
Drawer
2.8
Toolhar
2.2
Button Bar
0.4
(a) Rico
Table 23: AP changes Rico and DocGenome component types after InfoDet pre-traine pre-training.
L THE USE OF LARGE LANGUAGE MODELS
The use of LLMs in this work is limited to the following: 1) polishing the writing for clarity; 2) filtering candidate infographics during dataset construction.
32


Reviews:
[{"review_id": "6yhC3kAHp2", "reviewer": "Reviewer_fMwe", "summary": "This paper introduces InfoDet, a large-scale dataset for infographic element detection, designed to improve visual grounding and chart understanding in vision-language models (VLMs). The dataset consists of 11,264 real and 90,000 synthetic infographics, totaling over 14 million bounding box annotations covering texts, charts, human-recognizable objects (HROs), and sub-elements like bars and legends. The authors develop a hybrid annotation pipeline combining model-in-the-loop and programmatic methods to efficiently produce high-quality labels. Three key applications are demonstrated: (1) a Thinking-with-Boxes scheme that enhances grounded reasoning in VLMs , (2) benchmarking 11 object detection models on infographic data, and (3) transferring InfoDet-trained models to graphic layout detection in documents and UI design.", "questions": "How does annotation quality vary across infographic types (e.g., dense textual vs. image-heavy layouts)?\nCould the Thinking-with-Boxes approach extend to document reasoning tasks beyond charts (e.g., scientific figures, tables)?\nHave the authors considered releasing segmentation masks or relation graphs between detected elements to support compositional reasoning?\nCould the authors quantify the annotation process in terms of human effort and computational\nWhat are the computational costs of fine-tuning large detection models (e.g., Co-DETR) on InfoDet relative to standard datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper’s methodological rigor and practical significance are outstanding. The hybrid annotation strategy—combining synthetic and real data under a model-in-the-loop refinement—balances scalability and accuracy, achieving dataset quality comparable to COCO. The scale (over 100k infographics) and fine-grained annotations (charts, HROs, sub-elements) fill a clear research gap. The grounded CoT prompting demonstrates genuine insight into how structured visual grounding improves reasoning in modern VLMs. Empirical evaluations are extensive and carefully analyzed, revealing key insights (e.g., why zero-/few-shot detection fails on infographics). Finally, ethical considerations and licensing transparency are meticulously handled.", "weaknesses": "First, the paper lacks quantitative validation for synthetic data fidelity and bias, which is a critical limitation given that nearly 90% of InfoDet consists of synthetic infographics. Without quantitative analysis or cross-domain alignment metrics—such as style distribution, semantic bias, or embedding similarity—the fidelity of synthetic data relative to real samples remains uncertain, casting doubt on downstream generalization and fairness.\nSecond, there is insufficient analysis of annotation tradeoffs and reproducibility. Although the model-in-the-loop annotation pipeline is innovative, the authors do not quantify annotation cost, human effort, or refinement rounds, nor do they analyze how annotation quality affects detection accuracy, which weakens the dataset’s reproducibility and practical replicability.\nThird, the evaluation of the Thinking-with-Boxes method is limited in scope and validation. The experiments are restricted to a single benchmark (ChartQAPro) with relatively small gains (around 3–5%) and no statistical or qualitative error analysis, suggesting that the improvements may stem from prompt design rather than genuine reasoning enhancement.\nFinally, there is a lack of formal evaluation of annotation consistency and labeling noise. The reported annotation precision and recall are derived from only 1,250 samples and omit inter-annotator agreement or systematic error analysis. This absence of consistency validation leaves the internal reliability of the dataset uncertain and may affect the credibility of subsequent benchmarking results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2CTd9miOGf", "reviewer": "Reviewer_WrQm", "summary": "This paper presents InfoDet, a large-scale dataset for infographic-style element detection comprising about 101K images and 14M bounding boxes, annotated over text, charts, human-recognizable objects, and fine-grained chart components (26 mark types, 75 chart types). The authors argue that many failures of current multimodal/chart QA systems arise from inadequate grounding in cluttered infographic layouts, and they show that supplying detected elements to recent vision–language models improves performance on challenging multi-chart benchmarks.", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper targets a real and under-served pain point in multimodal/chart understanding—current VLMs and chart/infographic QA systems often fail not because they cannot “reason,” but because they cannot reliably ground the relevant regions in cluttered infographic-style inputs.\n\n2.The proposed InfoDet dataset is both large (≈101K images, ≈14M boxes) and unusually fine-grained, covering text, charts, and human-recognizable objects (HROs), as well as 26 chart-level marks and 75 chart types. This level of structural annotation is rare in existing public datasets and is directly useful for detection-then-reasoning pipelines.\n\n3.The “thinking-with-boxes” usage demonstrates that plugging the detected elements into strong VLMs yields consistent gains on challenging, multi-chart settings (e.g., ChartQAPro), which shows the dataset is not only a collection effort but can translate into practical improvements in multimodal QA.", "weaknesses": "1.The core novelty lies in building a large, high-quality dataset and a reasonable model-in-the-loop pipeline, plus a demonstrative prompting scheme. Compared to typical ICLR work, the methodological/learning novelty is modest.\n\n2. Given that the dataset provides structured and layered infographic elements, the paper could reasonably be expected to propose a model or training scheme that explicitly exploits this structure (for example, through element-level selection, layout-aware fusion, or hierarchical grounding). The current usage mainly demonstrates utility through prompting, rather than through a dedicated model design that fully leverages the annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57JjfiYfTO", "reviewer": "Reviewer_Jyhz", "summary": "This paper introduces InfoDet, a large-scale dataset for infographic element detection containing 11,264 real and 90,000 synthetic infographics with over 14 million bounding box annotations (texts, charts, HROs, sub-elements). Synthetic infographics are generated using 1,072 template-based methods, while real infographics are annotated via model-in-the-loop approach achieving 93.9% precision and 96.7% recall. The paper demonstrates usefulness through three applications: (1) Thinking-with-Boxes scheme improving o4-mini on ChartQAPro by 1.7pp, (2) comparing 11 object detection models showing foundation models struggle (DINO-X: 14.0 chart AP) while Co-DETR achieves 88.2 mAP after InfoDet fine-tuning, and (3) transfer learning on Rico and DocGenome showing InfoDet pre-training improves performance.", "questions": "**1. Dataset Quality: Synthetic Data and Annotation Process**\n- How does the 8:1 synthetic-to-real ratio with observed biases (hand-drawn style overrepresentation, missing composite charts) affect learned model behavior?\n- Why is fine-grained chart type annotation impossible for real infographics? Were better models (Gemini 2.5 pro, GPT5, Claude Sonnet 4) or human annotation attempted beyond GPT-4o's 61.49%?\n- How many expert annotators participated with what backgrounds and training? How many refinement rounds occurred with what precision/recall progression? What was initial auto-annotation quality and inter-annotator agreement, especially for ambiguous chart-HRO boundaries?\n\n**2. Application Mechanisms and Effectiveness**\n- For grounded CoT: Aren't bbox coordinates sufficient instead of full content text? Does current approach constitute \"answer leakage\" via OCR? How does bbox detection error (~12% given 88.2 AP) impact final QA performance - correlation analysis available?\n- Why minimal improvement on plain single charts - is native VLM visual reasoning already sufficient there?\n- For object detection: Does InfoDet fine-tuning cause regression on other domains (natural images)? What causes 6.5pp mark-level drop - class count, object size, or boundary ambiguity? Root cause of foundation model failure - simply pre-training data mismatch or intrinsic task difficulty?\n\n**3. Evaluation Rigor and Generalization**\n- Does grounded CoT generalize to other chart QA benchmarks (ChartQA, PlotQA)?\n- Does transfer learning generalize to other layout detection datasets beyond Rico/DocGenome?\n- Which specific component types in Rico/DocGenome benefit most from InfoDet pre-training, and why larger improvement on DocGenome?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1. Comprehensive and Well-Designed Dataset**\n- First large-scale infographic dataset (101,264 samples vs. prior Borkin et al. 393 samples) strategically combining real and synthetic data for authenticity and scalability\n- Efficient model-in-the-loop annotation achieving quality comparable to COCO (precision 93.9%, recall 96.7% vs. COCO's 71.9%/83.0%)\n- Multi-level annotations: element-level (charts, HROs) and mark-level (26 sub-element categories) providing fine-grained labels\n- Verified diversity across 67 chart types, 96 topics, 15 visual styles (Appendix F)\n\n**2. Important Empirical Findings on Model Capabilities**\n- **Foundation model limitations**: State-of-the-art models like DINO-X achieve only 14.0 chart AP and 15.0 HRO AP in zero-shot, with few-shot prompting showing minimal improvement (MQ-GLIP 4-shot: 16.2 chart AP). This reveals natural scene pre-training is insufficient for graphics domain.\n- **Fine-tuning effectiveness**: Co-DETR improves dramatically from 27.6 AP (4-shot) to 88.2 AP (InfoDet training), demonstrating value of large-scale quality data\n- **Generalization capability**: Cross-dataset evaluation shows InfoDet -> VG-DCU transfer drops only 17.1% mAP vs. 53.7% drop in reverse direction, indicating infographic data learns more general representations than plain charts\n\n**3. Thorough Experimental Protocol and Transparent Reporting**\n- Standardized training protocol ensuring fair comparison across 11 models\n- Ablation studies demonstrating complementary roles of visual and textual prompts (64.1 combined vs. 62.8 visual-only, 61.6 textual-only)\n- Transfer learning validated on Rico (42.1 -> 50.6 -> 53.6) and DocGenome (69.0 -> 74.4 -> 80.0)\n- Honest reporting of limitations (minimal improvement on plain single charts, o3's instruction-following issues)", "weaknesses": "**1. Dataset Construction Issues: Representativeness and Transparency**\n- **Fine-grained annotation imbalance**: 75 chart types exist only for synthetic infographics. Authors mention GPT-4o achieved only 61.49% accuracy on real infographics but provide no alternative approach (human annotation? better models?), leaving dataset incomplete.\n- **Annotation process opaque**: No information on expert demographics (number? background: medical imaging experts? graphic designers? CV researchers?), training process, or inter-annotator agreement. \"Multiple rounds of refinement\" lacks specifics: how many rounds? How did precision/recall improve each round? What was initial auto-annotation quality? These details are critical for reproducibility and trustworthiness.\n\n**2. Limited Application Improvements with Insufficient Analysis**\n- **Grounded CoT modest gains**: Overall 1.7pp improvement is limited, especially negligible on plain single charts (58.1→60.6) with clear improvement only on infographic multiple (70.6→72.5), suggesting method effectiveness is confined to complex scenarios.\n- **Detection evaluation lacks depth**: No analysis of post-fine-tuning regression on other scenarios (natural images). Mark-level performance drops 6.5pp from element-level (76.1% -> 69.6%) without investigating causes (more classes? smaller objects? ambiguous boundaries?). Chart detection improves 3.2× (27.6 -> 88.2) while HRO only 2.5× (25.5 -> 64.0) - why is HRO harder? Which HRO types (data-related vs. theme-related) are particularly challenging?\n\n**3. Insufficient Motivation and Missing Broader Context**\n- **Limited practical justification**: Introduction only states \"charts are fundamental\" without explaining why infographic understanding matters or concrete use cases (e.g., social media analysis, news understanding, educational accessibility, automated fact-checking for visually impaired users). Lack of real-world deployment scenarios weakens motivation.\n- **Limited generalization evaluation**: Grounded CoT only tested on ChartQAPro - does it work on other chart QA benchmarks (ChartQA, PlotQA)? Transfer learning only on Rico/DocGenome - what about other layout detection datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iqg3QKiYy3", "reviewer": "Reviewer_x562", "summary": "The paper introduces InfoDet, a large-scale dataset for detecting infographic elements， including charts, text, and human-recognizable objects (HROs), to improve visual grounding for chart understanding. InfoDet combines 11,264 real and 90,000 synthetic infographics with over 14 million bounding-box annotations, produced via a hybrid programmatic + model-in-the-loop pipeline. The paper demonstrates InfoDet's value via three applications: (1) enabling a \"Thinking-with-Boxes\" (Grounded CoT) scheme that significantly improves VLM performance on the ChartQAPro benchmark (+49.6 Enhanced Relaxed Acc. for O4-mini, Fig 1); (2) benchmarking object detection models, showing InfoDet-trained models outperform SOTA foundation models ; and (3) showing the InfoDet-trained detector generalizes to UI and document layout tasks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work introduces InfoDet, a large-scale dataset featuring 101k images and 14M annotations, specifically designed to address the complex layouts of infographics that integrate charts and Human-Related Objects (HROs).\n2. The usefulness of the dataset is validated across diverse downstream tasks (Section 4). The authors provide compelling quantitative evidence for its impact on VLM grounding (+49.6 gain on ChartQAPro, Sec 4.1), its utility as a challenging benchmark for object detectors (Table 4), and the robust generalizability of its features to out-of-domain tasks (Table 5)..", "weaknesses": "1. The model-in-the-loop method (Sec 3.2) may introduce systematic biases from the initial model (trained on synthetic data) into the real-image annotations. Please discuss this limitation and ideally quantify it, perhaps by reporting Inter-Annotator Agreement (IAA) among experts or by comparing against a small, fully-manual gold-standard set.\n2. The method layers visual overlays and textual descriptors, but it’s unclear how much gain comes from (i) two-layer overlays, (ii) textual element lists, or (iii) better detections/OCR. Please include controlled ablations that independently remove the text lists, the two-layer separation, and vary detector/OCR quality.\n3. The paper does not include a dedicated limitations discussion. A concise reflection on dataset scope and failure modes would improve transparency and guide responsible use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)