BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks
Authors: Juan A. Rodriguez
Xiangru Jian
Siba Smarak Panigrahi
Tianyu Zhang
Aarash Feizi
Abhay Puri
Akshay Kalkunte Suresh
François Savard
Ahmed Masry
Shravan Nayak
Rabiul Awal
Mahsa Massoud
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang
Pierre-Andre Noel
Mats Leon Richter
Saverio Vadacchino
Shubham Agarwal
Sanket Biswas
Sara Shanian
Ying Zhang
Sathwik Tejaswi Madhusudhan
Joao Monteiro
Krishnamurthy Dj Dvijotham
Torsten Scholak
Nicolas Chapados
Sepideh Kharaghani
Sean Hughes
M. Özsu
Siva Reddy
Marco Pedersoli
Yoshua Bengio
Christopher Pal
Issam H. Laradji
Spandana Gella
Perouz Taslakian
David Vazquez
Sai Rajeswar
Venue: ICLR 2025 Poster (2025)
Keywords: datasets
vision language models
multimodal
large language models
Primary area: foundation or frontier models, including LLMs
TL;DR: BigDocs is a large-scale, license-permissive dataset for continual pretraining on document understanding tasks. We also introduce 9 benchmarks to assess structured output generation, like code generation and reasoning from documents

Abstract: Multimodal AI has the potential to significantly enhance document-understanding tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long-structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access to relevant training data and restrictive licensing, which hinders open access. To address these limitations, we introduce BigDocs-7.5M, a high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process to ensure that our data is high quality and license-permissive. Our process emphasizes accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs-Bench,, a benchmark suite with 10 novel tasks where we carefully create datasets that reflect real-world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDocs-Bench, improves average performance up to 25.8% over closed-source GPT-4o in document reasoning and structured output tasks such as Screenshot2HTML or Image2Latex generation. Finally, human evaluations revealed that participants preferred the outputs from models trained with BigDocs over those from GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve AI tools to enhance multimodal capabilities and document reasoning.

Published as a conference paper at ICLR 2025
BIGDOCS: AN OPEN DATASET FOR TRAINING MULTI- MODAL MODELS ON DOCUMENT AND CODE TASKS
Juan Rodriguez 1,2,3,a 1,2,,
Xiangru Jian 1,4aa
14,a
Siba Smarak Panigrahi
Tianyu Zhang!.2.5. 1,2,5,a
Aarash Feizi
Abhay Puri
Akshay Kalkunte 1.
François Savard 1.a
Ahmed Masry!.7h
Shravan Nayak!,20 1,2,5,b
Rabiul Awall.2.5. Mahsa Massoud 1,2,6,h
Amirhossein Abaskohi
Zichao Li
Suyuchen Wang2.5,
Pierre-André No
Mats Leon Richter
Saverio Vadacchino'
Shubham Agarwal'.2 Sanket Biswas
Sara Shanian"
Ying Zhang'
Noah Bolger'
Kurt MacDonald Simon Fauvel!
Sathwik Tejaswi!
Srinivas Sunkara' Joao Monteiro! Krishnamurthy DJ Dvijotham
Torsten Scholak!
Nicolas Chapados!
Sepidch Kharagani*
Sean Hughes! M. Özsu*
Siva Reddy1.2.6.0
Marco Pedersoli
Yoshua Bengio
Christopher Pa11.2.10.11
Issam Laradji
Spandana Gella!
Perouz Taslakian
David Vazquez!
Sai Rajeswar
"ServiceNow
MMila
"Licole Technologic Supérieure
Uniieesity of Waterloo
Universstté Montrëal
MMeGill University
York University University of British Columhia
"Universitat Autònoma Barcelona
"CIFAR Chair
" "Polytechnique Montréal
"Finst Author Equal contribution.
"Second Author Equal contribution.
ABSTRACT
Multimodal AI has the potential to significantly enhance undieraannn tasks, such as processing receipts, understanding workflows, extracting data from documents, and summarizing reports. Code generation tasks that require long- structured outputs can also be enhanced by multimodality. Despite this, their use in commercial applications is often limited due to limited access training data and restrictive licensing, which hinders open access. To address these limitations, we in- troduce BigDocs-7.5M, high-quality, open-access dataset comprising 7.5 million multimodal documents across 30 tasks. We use an efficient data curation process ensure our data is high figh-quaiity and accountability, responsibility, and transparency through filtering rules, traceable metadata, and careful content analysis. Additionally, we introduce BigDocs- Bench, benchmark suite with 10 novel tasks where create datasets that reflect real- world use cases involving reasoning over Graphical User Interfaces (GUI) and code generation from images. Our experiments show that training with BigDoes- Bench improves average performance up to 25.8% over closed-source GPTT-4o in document reasoning and structured output tasks such Screenshot2HTMLL or Image2Latex generation. Finally, human evaluations showed a preference for outputs from models trained on BigDocs over GPT-4o. This suggests that BigDocs can help both academics and the open-source community utilize and improve Al tools enhance multimodal capabilities and document reasoning. The project hosted at https: /ttp///si...//// github. io.
Our process emphasizes
INTRODUCTION
Visually-Rich Document data containing text and structured elements (such as charts, infographies, diagrams, sketches, tables, etc.) are essential cues for users to efficiently understand complex information holistically. To facilitate this understanding, foundation models must process these structured documents and subsequently extract key insights, identify patterns, and generate concise summaries and responses from user requests with reasoning (Landeghem et al., 2023; Zhu et al., 2022). Recent advances in multimodal AI (Yang al., 2023; Team et al., 2024b) have demonstrated impressive capabilities, including generating functional web pages (Zheng et al., 2024), automating document understanding workflows (Wang et al., 2023b), and extracting detailed information from

---

Published as a conference paper at ICLR 2025
CORD Purk (33 K)
ToeOCR S.boes (2000 CIMP-IM" (2985.0
COCO-Teu (47.0
OUR-VQA
(1019)
AborCtie (416.
Strueture
Mareaan fn
DooBark Li (2000)) (4000))
TalkBank Li al (20001) K) QUIUUerraee (214)
vntte
OCR selati
oke Ges
Captionin
Captioning
DORD
OpesBosiess Singh et (2000)
SVGCip Rodrgaes et (20134)-1000
an
Sinicture
Code
ChanEMakönag'
TaNtis Char (2000) (38.5 K)
AnicTiaaao Arkess (305)) (196 2.)
ChacCCaptinl (8.0
Lnand it
OOCA
Della
WAkTT) Barpea Liung (201 (213 KJ Kibos (2007) (800
UniChert Manyy (2023) (6550 GUB VQA Cheng (3001) (90.1k
SpersskedirES (11.3
Imnge22foo (20.0 X)
TTNNAT Arira (76.5
150g Recrigaer et (30770) (3195K)
Figure 1: BigDocs: A Scaee-Scale Structured Continual Pretraining and Finetuning Dataset. The inner circle represents the distribution of BigDocs, derailing the categories. The outer circle displays the specific datasets compiled to form 7 million image-text pairs. Dutasets with * denotes our contribution.
documents to produce comprehensive reports (Borchmann, 2024). However, the datasets used to train these models remain closed-source with undisclosed details and restrictive licensing, which hampers their broader adoption in research and the advancement of open-source model development. In contrast, current open-source models and datasets (Chen et al., 2023; Liu et al., 2023b; 2024) primarily focus on basic document understanding tasks, such as optical character recognition (OCR), e.g., DocStruct4M (Hu al., 2024), basic question-answering and mathematical problems, e.g.. Cambrian 7M (Tong et al., 2024). These efforts do not sufficiently address the complexity of processing intricate visual documents or generating long- sangsstructured outputs, such as JSON and HTML, which are valuable in real-world applications.
In this work, we present BigDoes (https: /biggocss github. 1o), a large-scale and open dataset, benchmark suite, and models specifically designed for user-facing document-related tasks. BigDocs bridges existing gaps by enabling open-source models to meet the rising demand for so- phisticated document understanding technologies. Comprising 7.5 million image-text pairs, BigDocs carefully curated to support three core areas: (I) Document Information Extraction, which includes enhanced OCR for diverse document types, named entity recognition, layout analysis, and table detection; (2) Document Understanding, covering semantic comprehension tasks such as document classification, question answering, and analysis of diagrams; and (3) Document Creation and Manipulation, converting visual data into structured formats like HTML, La'TeX, and JSON.
Our survey of 133 existing datasets revealed that 80% of them (i.e., around 100 datasets) have either clear licensing informa- tion (Chaudhry et al., 2019; kleister Charity, 2021). creating barriers to reuse and transparency. In response, BigDocs prioritize datasetss with permissive licenses (e.g., CC-BY-4.0, MIT) and document- related information, ultimately retaining 16 fully accessible datasets. To further support accessibility, we developed the BigDocs Toolkit, which offers modular tools for data preprocessing, filtering, and consolidation. Additionally, we introduced unified metadata framework to enhance dataset traceability (e.g., properties, sources, licenses), including detailed documentation of transformations applied by us to the original data. We also conduct a data contamination analysis on downstream tasks data, showing that BigDoes- 7.5M has lower contamination rates than previous datasets.
non-errmissive licenses (Jaune et al., 2019; Stěpán Šimsa et al., 2023)
To further advance document intelligence, BigDocs- BigDocs-Bench offers 10 novel downstream tasks, each with four splits: train, validation, test, and hidden test (with 329k training samples, 11k validation samples, 10k testing samples). These tasks focus on structured output generation, including code formats such as HTML, LaTeX, SVG, and Markdown. Our experimental results demonstrate that models trained on the BigDocs suite outperform those trained existing datasets like DocStruct4M (Hu et al., 2024) on standard document benchmarks. Additionally, automatic and human evaluations on the novel tasks introduced in BigDocs Bench highlight the advanced capabilities of these models in generating format, structured outputs. User evaluations reveal a preference for our models' outputs 88% of the time over Phi3.5 Instruct and 63% over GPT-4.
2

---

Published as a conference paper at ICLR 2025
License Filtering Comerctial Permistre Datorets
Crawled Dataset Sources with Permisstre Licenses
Documment Datasets
Curation propused HigDory
CDIP
CDIP
Araiv PDF OCR
1. OCR Extraction HTMMLSVGMMI curation
Inschart-Table
Unichart-Table
Diagrams, Tables Latex
Content filtering VQA format
Websighi2
Websight2
x
HTML Code
5. Text-Image Aligament
DocViA
DocVQA
x
Slides & OCR
InfogrsphicVQA
InfogrophieVQ4
Flowcharts
OCRVVQA
OCR-VQA
Scalable Vector Graphics
BigDocs-7.5M
137 datervts
poblic denssets offered perridssted Roeesse
Teas and Code extracted froms per crrmiseee webbette academiic papers
conssdernd
Figure 2: BigDocs-7.5M Dataset Curation. The figure illustrates the extraction, filtering, and curation process of BigDocs-7.5M, which emphasizes maintaining permissive licensing. To build BigDocs-7.5M. we first gather publicly available vision- lsion-language datasets, particularly those centered on document analysis, and apply rigorous filtering process. We then augment these datasets with crawled data. Finally, standardize all samples and tasks into a unified format produce BigDocs-7.5M
Built with commitment to accountability, responsibility, and transparency (ART) (Bommasani al., 2023; Vogus & Llansóe, 2021), BigDocs is open-sourced, including datasets, models, and documentation to foster responsible Al development. In summary, BigDocs contributions include:
B.......7.5M, a large-scale, picense-permissive dataset designed for continual pretraining (further training from a pretrained foundation model checkpoint) and downstream finetuning (e.g.. to follow instructions or task formats) of multimodal models on document-relatee tasks. It includes traceable metadata and curated licensing drawing from document-rich multiple data sources, ensuring full public accessibility.
2. SigDocs-Bench, set of 10 new benchmarks, including test datasets well corresponding innovative evaluation metrics for multimodal models to generate long-structured code outputs from images, including formats such as HTML, LaTeX, Markdown, and SVG.
3. BigDocs Toolkit, unified tools supporting open-source efforts. These tools allow efficient data curation, filtering, formatting, and preparation for training models generating structured outputs.
4. BigDocs Models: We conduct extensive experiments using four state-of-the-ar public models, demonstrating the advantages training with BigDocs over alternative datasets and enabling the models to learn novel tasks through our dataset suite.
2 RELATED WORK
Multimodal Datasets. General-purpose vision-languaal datasets like COCO Caption (Chen et al., 2015) and SBUCaption (Ordonez et al., 2011) primarily feature photographic content, lacking visually-rich document (VRD) data (Sharma et al., 2018; Changpinyo et al., 2021). In contrast, our focus is on text-heavy datasets including PDFs, tables, and invoices (Veit et al., 2016; Mishra et al.. 2019; Singh et al., 2021; Li et al., 2020b;a; Soboro, 2022), which crucial for tasks like information extraction and parsing (Masry et al., 2022; Rodriguez et al., 2023c;b). While datasets like DocStruct4M (Hu et al., 2024) and Cambrian7M (Tong al., 2024) partially address these needs, they often lack permissive licenses or are not open-source. Kosmos-2.5 (Ly et al., 2023) focuses on building large document dataset; however, the authors did make it public. BigDocs fills these gaps by providing 7.5M permissively licensed image-text pairs from 16 academic datasets and other open platforms, supporting diverse document understanding tasks.
Responsible Data and Licensing. Enterprise models like GPT-4 (OpenAI, 2023) and Claude (An- thropic, 2024) are often closed-source, lacking transparency in training data (Bommasani et al., 2024). Foundational works (Gebru et al., 2021; Bender & Friedman, 2018) emphasize the importance of open access and transparency in dataset documentation. Previous works promoting open-access such as StarCoder (Li et al., 2023), The Stack (Kocetkov et al., 2022), FineWeb (Penedo et al., 2024), and LLaMA (Dubey et al., 2024), have addressed this by releasing data or models for language models pretraining. Our work builds these efforts by creating curated, well-documented resource for
3

---

Published as a conference paper at ICLR 2025
open access (Laurençon et al., 2023). addressing licensing complexities ranging from permissive licenses (c.g., Apache 2.0, MIT (Hu et al., 2024; Gadre et al., 2023)) to restrictive ones like CC BY-SA (Foundation, 2024: Zhang et al., 2024) and CC BY-NC-SA (Wang et al., 2024b). Compound datasets, like Cambrian-7M (Tong et al., 2024), face mixed licensing issues, while some datasets, such as VOA (Mathew al., 2021b), have different licenses for images and annotations. In developing BigDocs, we prioritized permissive licensing in data curation and contributed to new datasets that adhere to open-access principles, ensuring transparency all stages of development.
Multimodal Document Understanding Models. Recent advancements have introduced several general-purpose multimodal models (Liu et al., 2023b;a; 2024; Bai et al., 2023b; Laurençon et al., 2024; Tong et al., 2024). For example, LLaVA (Liu et al., 2023b) integrates a vision encoder with a language model, with later versions enhancing reasoning and OCR capabilities (Liu et al., 2023a; 2024). Qwen2-VIL LLL (Bai et al., 2023b) processes images at native resolutions, while Phi 3.5 Vision (Abdin et al., 2024) offers a lightweight model for reasoning tasks. Specialized models for visually-rich documents, like DocOwi1.5 (Hu al., 2024) and DocLLM (Wang et al., 2023a), have also gained traction, particularly for commercial applications. However, the datasets for training these models are often not publicly available have restrictive licenses (Hu et al., 2024). Our work, BigDocs, addresses this by consolidating existing permissive datasets to support the development of reproducible, commercially viable document understanding models.
BIGDocS-7.5t1
BigDocs-7.5MM is large-scale, ficense-permissiie and carefully curated dataset for visual document understanding designed to train foundational models across various document types and tasks. consolidates public datasets and newly crawled data with permissive licenses by preprocessing. cleaning, and filtering them into a unified collection of 7.5 million image-text pairs. All curated datasets and related artifacts will be openly released foster community collaboration (Bender & Friedman, 2018). The curation process is illustrated in Figure 2 and detailed below.
3.1 DATASET CURATION PROCESS
Existing Dataset Acquisition. The authors, along with domain experts and researchers, guided the collection strategy, assessing dataset relevance, quality, and diversity. We gathered 133 public vision-language datasets by searching academic repositories, open data platforms, and research papers. The collection focused tasks like image captioning (Chen et al., 2015; Sidorov et al., 2020), OCR (Park et al., 2019; Smock et al., 2022), visual question answering (Mishra et al., 2019; Mathew et al., 2021b), scene-text recognition (Veit et al., 2016; Singh et al., 2021), and document layout analysis (Li et al., 2020a;b), resulting in a diverse multimodal dataset repository.
Datasheets for Datasets. During data acquisition, we compiled detailed datasheets for each dataset, capturing metadata such as ownership, status, size, references, source type, annotations, licensing, and specific observations. We filtered datasets based on licensing compatibility and relevance to our document-related tasks, then extended the datasheets of selected datasets for better categorization. This extension organized datasets by attributes such as medium type (e.g., digital, scanned), document type (c.g., articles, infographics). sourcing method, text type (e.g., computer-generatel handwritten), structure, language, timeframe, and licensing. For licensing, we documented both the image licenses and annotation licenses separately, as these often differed and impacted the overall permissiveness and usability of each dataset. This structured approach aligned datasets with specific use cases (e.g.. OCR, structured parsing) and grouped them for pretraining, finetuning, and evaluation, ensuring effective integration into our visual document understanding pipeline.
License Filtering, A key criterion for dataset selection ensuring permissive licenses (c.g., CC-BY, MIT, Apache 2. CCO) for both images and annotations, suitable for open access and commercial use (more details on various of licenses in Appendix A.7). Datasets with non-permissive licenses, like DVQA (Kafle et al., 2018) (CC-BY-NC 4.0) or DocILE (Štěpán Šimsa et al., 2023) (non-commercial use), or with no license information, like DeepForm (Svetlichnaya, 2020), were excluded. Ultimately, prioritized permissive licenses for both text and images, resulting in 20% being kept, while 7.5% moderately restrictive and 72.5% non permissivee were discarded. Some included datasets still have images under less clear terms, such as "Fair Use" (e.g., OCR-VQA), documented in the metadata.
4

---

Published as a conference paper at ICLR 2025
KLC
Threshold 0.96
Threshold 0.97
Threshold 0.98
Figure 3: Assessing data contamination (smaller is better). The radial axis (log scale) indicates the proportion of images from the evaluation dataset that exhibit similarity to a training sample beyond given threshold according CLIP. Human evaluations indicate that instances captured at threshold of 0.98 are prob- lematic, and most problematic samples are identified at a threshold of 0.96. Except for MMMU and DudeMini. BigDocs-7.5M (blue/darkred) less contaminated compared to DocStruct4M (red).
3.2 BIGDOcs TOOLKIT: DATA PREPROCESSING, FILTERING, AND CONSOLIDATION
The BigDocs Toolkit provides modular tools for preprocessing, filtering, contamination management, metadata management, and dataset loading. These components work in unison streamline the integration of large-scale document datasets, ensuring quality and ease for efficient model training.
Datamaker Module. The BigDocs Toolkit offers a modular framework for dataset curation, focusing on standardization, quality control, and metadata management. Its core DataMaker class acts as a template for handlers that extract annotations and convert raw data into a standardized format. universal funetion processes tasks like OCR, VQA, and code generation, ensuring consistency. Bounding boxes are standardized, and corrupted samples are filtered. The Toolkit also generates metadata enhance transparency, covering licensing and processing details (see Appendix A.9).
@ Unified Metadata Framework. We propose a unified metadata framework for Big to ensure transparency and traceability. This framework thoroughly examines each raw data source, extracts fine-grained license information, and documents transformations applied to the data (c.g., different sources may have distinet licenses). Fach data sample includes metadata attribute detailing its properties, licenses, sources, and transformations (see Appendix A.9 for an example in Figure 13 and structure details). To our knowledge, this the first systematic approach to track metadata for visually rich documents, advancing transparency in multimodal dataset curation.
Assessing Contamination. The presence of downstream evaluation data in training datasets can significantly affect the accurate measurement of model's effectiveness (Magar & Schwartz, 2022). BigDocs-7.5MM contains samples from the training split of TabFact, WTQ, and TextVQA. However, any evaluation dataset may a priori overlap with training datasets through less direct dependencies. We favor transparency strategy: search for overlaps and report what was found. Figure 3 gives an overview the main observations on this front; see Appendix A.3 for additional details.
3.2.1 THE RESULTING BIGDoCS-7.5M
BigDocs- consists of 7.5M image-text pairs for training (4M unique images), 500k for validation (234k unique images), and 470k for testing (261k unique images). These data points aggregate four enabling
document-related tasks OCR, structured parsing, captioning, and question-answerdin models to handle diverse document use cases (see Figure and Appendix A and A.1 for more details). Low-quality data filtered out, and metadata details our best-effort licensing compliance, making the dataset suitable for training foundational models, even in commercial applications.
4 BUILDING TIGDoCS-BENCE
In the previous section, introduced 7.5 unified and permissive dataset for training models on document understanding tasks. Building on this, we present BigDocs-Beenhh a benchmark suite for evaluating downstream tasks that transform visual inputs into structured outputs, such as
5

---

Published as a conference paper at ICLR 2025
Table 1: Statistics of the ten downstream tasks in BigDocs-Bench GPT2 tokenizer is used to produce token numbers of both queries and annotations (if any), where the format is (avg + sid).
Downstream Task
TTriiinn
# Validation
# Public Test
Hidden Test
# Text Tokens
ScreenshoQ2ITMM.
9,338
1,000
500
500
32,700 1. 53,105
X
Table2LaTeX
77.669
1,000
500
500
438 +: 540
Image2SVG
198,000
2,000
748
748
2,871 1: 1,728
Image2Flow
8,000
1,000
500
500
418 + 124
Image:2Flo (SON)
8,000
1,000
500
500
1,771 + 601
M1
Markkoown
4,516
1,000
500
500
1,559 1 4,442
Chart2Caption
5,412
1,300
650
650
94 + 49
seilUnern
79,000
1,000
500
500
28 1
GUI2Summary
79,000
1,000
500
500
132 25
GUI-VQA
78,991
1,000
500
500
24
Screensbot23IMM
ImageZElows
Tablellatex
Image/Tes125V
GIT-IAA
Grn25umm
Chart&Captinn
Figure 4: of the new tasks introduced in Big BigDocx-Bench. Bench. These tasks share a focus on understanding the underlying structure of visually rich documents, with many also requiring generating lengthy oulputs, such as SVG and HTML code. More tasks are shown in Figure 9 and
GUI2UserIntent (fine-graines reasoning), Image2Flow (structured output), Chart2Caption (under- standing), and Screensho22HTML. (creative generation). BigDocs-Bend includes ten specialized tasks, with 329k training samples, 11k validation samples, 10k testing samples, and an additional hidden test set. Refer to Table for task details and Figure for examples.
4.1 MIGDOCS-BENCL TASKS SUITE
</> Screensho12HTMI: We introduce benchmark for Screenshot2HTMM.. conversion, with 10,838 real-world website screenshots paired with HTML code (see A.4.2). Curated from diverse, text-heavy websites in the FineWeb corpus (Penedo al., 2024), it contrasts with synthetic sites from prior work (Laurençon et al., 2024b). Using P Playwright, we retrieved, rendered, and filtered sites for accessibility, content, and licensing. External assets (e.g.. CSS, fonts) were inlined, JavaScript removed, and images replaced to focus on structure. Screenshot2HTML evaluates the accuracy of HTML generated from webpage sereenshots, emphasizing layout fidelity and semantic correctness.
X Table2LaTeX::
We propose a benchmark for Table2LaTeX conversion, consisting of 79,669 table images paired with original LaTeX code and captions (details in Appendix A.4.3). The dataset was curated by crawling arXiv papers with permissible licenses and extracting tables from PDFs and TeX source files(ht : / //axxivv... org/). Instead of relying on imperfect PDF detection, tables were rendered from the LaTeX tex code to ensure accurate visuals. For instance, an image of table is paired with its corresponding LaTeX snippet code and caption. This benchmark supports precise table extraction and LaTeX generation evaluation from academic documents.
Image2SVG: We present a benchmark for Image2SVG conversion, curated from the existing SVG-Stack collection by Vtator (Rodriguez et al., 2023a) (details in Appendix A.4.4). The dataset includes 200k raster images paired with SVG code (e.g.. flowchart image paired with SVG code replicating it) or descriptive text. We filtered for complex designs. using image entropy to exclude simple graphies, and ranked image-text pairs by CL.IP Score (Hessel et al., 2022; Radford et al., 2019). This benchmark evaluates models on precise vector image reconstruction and scalability.
Image2Flow: We introduce two benchmarks, Image2Flow and Image2Flow mapping flowchart images to JSON or GraphViz code (Appendix A.4.1). The dataset includes 10,000 flowchart samples with GraphViz files generated using LLaMA 3.1 (Dubey et al., 2024) and ISON files detailing nodes and connections. Random colors and styles were applied for diversity.

---

Published as a conference paper at ICLR 2025
Unlike FlowchartQA (Tannert et al., 2023), which focuses QA over preprocessed flowcharts, this benchmark tests models' ability to extract structure from raw images.
This dataset assesses the models' capabilities in extracting data values from chart images: given an image. produce a data table of the underlying data table in markdown format. To create this dataset, crawled recent chart images from the Statista website (ht Inttps: com/), ), focusing on charts from 2023 and 2024 that were not used in prior datasets like UniChart (Masry et al., 2023) and ChartQA (Masry et al., 2022). Our dataset reflects the most up-to-date facts and trends and overlaps less with existing datasets and benchmarks. We collected 6,516 chart images with their data tables and human written summaries.
M4
Chart2Markdown:
wtw.wsstati...
Char12Caption: We introduce a benchmark for Char12Caption conversion, aiming to generate textual summaries from chart images. The dataset includes 6,516 samples, with charts sourced from Kaggle by running public analytics notebooks and extracting charts and code. Summaries were generated using multimodal InterVL.2-26B model (Chen et al., 2023; 2024) based on a custom prompt (see Appendix A.4.5) and augmented with human-written summaries from the Chart2Markdown task. This benchmark evaluates models' abilities to interpret and summarize visual data representations.
This benchmark interprets user intent from GUT interactions, identifying elements linked to clicked bounding boxes (details in Appendix A.4.7). The dataset, repurposed from SeeClick (Cheng al., 2024), includes 80,000 website screenshots with bounding box coordinates and corresponding user intents sourced from Common Crawl to capture user interactions effectively.
GUItUserIntent:
The GUI2Summary task generates descriptions of website screenshots, focusing on web layouts. We synthesized 80,000 summaries (under 100 words each) using Intern 8B (Chen et al., 2023) in zero-shot setting (details in Appendix A.4.8). Each summary provides an overview of the main content, referencing key visual elements, layout, and color schemes.
GUI2Summar::
GUI-VQA: The GUI-VQA task answers questions about website screenshots, focusing on content and elements. We generated 80k QA pairs using sentences from GUI2Summary and prompting LLaMA 3.1-8b (Dubey al., 2024) in zero-shot setting (details in Appendix A.4.9).
4.2 FILTERING AND QUALITY
T Filtering and Verification with BigDocs Toolkit. To ensure high-quality, open-access dataset, we employed an NSFW detector (e.g.. Llama-Guard-3) and filtering tools to eliminate harmful content, corrppted images, misaligned annotations, and personally identifiable information (PII). The BigDocs Toolkit, adhering to ART principles, streamlined web crawling and filtering processes. This multi-layered approach, iteratively refined using typical errors identified during human verification, ensured a clean and reliable dataset. The test set underwent manual human verification, with at least two annotators per sample to ensure quality and accuracy. Sampling-based checks on the training split further validated the robustness of the filtering process, achieving a 99% pass rate (see A. A3).).
5 TRAINING MULTIMODAL MODELS ON BIGDOCS
We trained several state-of-the-ar multimodal models of varying sizes and architectures on BigDocs assess its effectiveness for document-based continual pretraining and downstream finetuning. For comparison, we also trained the models on DocStruct4M, the closest alternative in terms of scale and document-oriented tasks. Additionally, we experiment on the training set from BigDocs-Bench that requires generating long structure outputs, e.g., valid code outputs.
We follow two stages of training: continual pretraining (CPT) and downstream finetuning (FT). The datasets, such as BigDocs and DocStruct4M,
CPT stage involves training on large learning general tasks like OCR, layout understanding, and captioning. For FT smaller datasets, such as DocDownStream and BigDocs-Benie Bench, to focus on specific tasks like question answering or generating HTML from images. A previous stage of pretraining (PT) is typically performed for general multimodal alignment. In our framework, we do not perform this stage and rely on publicly available checkpoints. While pretraining (PT) is typically performed for general multimodal alignment, we rely public checkpoints instead. In CPT, we train the image encoder and connector to align image features with the LLM. For FT, both the connector and LLM remain unfrozen. See Figure 11 in Appendix A.5 for more details.
7

---

Published as a conference paper at ICLR 2025
Table 2: General Document Benchmarks. Models trained on (BigDocs-.33M DooeDDDDDDwwr Downstrar))) perform competitively across nultimodal document benchmarks. We compare them to base checkpoints and those trained on (DocStruct4M+ DDocSrnccA--DDeDoosstrrrer DocDownstream) Results for instruction-ture models (first row of each block; italic) are provided for reference only (unfair comparison). BigDocs models show consistent performance.
SlideVgA-M
DacV
Infa wnc.
Form
TabFact
Cbartih
TouVON
MMMU
DudeMial
Table 18l1
Arg. Score
KLC
WTO
man
Model
DocOwll..588 (instract)
80.73
69.94
68.31
37.99
38.87
79.67
68.55
68.91
33.67
36.64
31,62
52.60
53.54
1.5-8 (base)
2,07
1.84
0.00
0.00
0.00
0.00
0.00
0.00
24.44
19.07
3.30
13.63
5.36
DocDuil15 5-88 (bave) DocStruccM
75.99
46.88
62.77
35.21
32.86
71.56
68.36
65.00
33.67
29.00
27.03
46.27
49.56
DecOwil.5-50
BigDocs (Ous)
78.70
47.62
36.93
72.65
67.30
32.33
32.55
Owenz: VL. 28 (mstrrur)
89.16
64.11
72.35
25.75
38.20
37.21
73:40
79:
32,00
43.23
46.50
43.07
33.00
Qwen2-N1 VL 2B (base)
7.26
0.78
0.00
0.00
0.00
0.00
0.00
1.14
34.89
28.43
14.55
0.00
7.25
Qwen2 VL- 2B (buse) DoeStraceM
59.53
32,00
53.98
36.38
28.18
61.24
51.44
55.
34.89
28.78
22.68
46.53
43.15
Owen2-VL-2BB (base) a BigDoes (Ours)
57,23
31.88
49.31
34.39
31.61
64.75
68.60
61.01
35.67
27.19
47.53
PAGt 5- Visiooon-4f (instract)
86.00
56.20
10.47
7.49
17. FM
30:43
K2. 16
74. F2
46 01
57.20
30.99
711 71
45:65
MisiinV DocSSiacc4M
86.76
69.90
70.12
37,83
$1.30
82.12
79.76
68 60
24,11
35.52
31.90
69.17
60.51
Phi3.S. Vision-4B 4B BigDoes (Ours)
87.05
70.05
70.97
37,45
51.21
81
81.56
68.72
45.00
36.15
32.47
67.77
60.80
TaVV--WeX--7 (instrurt)
61:37
10330
7:30
3.35
207005
12.03
32:72
35.89
17.94
7:40
12:87
32.
LLaVA LLaVA-NeXT-77 DocStrucrM
60,95
26.14
39.78
28.31
25.
67.72
61.20
52.25
25.78
21.70
15.33
27.03
37.68
LLVAA-NXT--78- BigDoes (Ours)
57.13
24.47
46.38
31,09
27.06
72.58
54.72
49.06
17.78
22,88
16.07
33.13
37,70
5.1 EXPERIMENTAL SETUP
Baseline Models. We selected DocOw11.5-8B (Hu et al., 2024), Qwen2VL-2B (Bai et al., 2023a), (Abdin et al., 2024), and LLaVa-NeXT-7B (Li et al., 2024) for our training experiments. These models were chosen due to their focus on document-related tasks (DocOwl1.5), their openness regarding checkpoints (Qwen2VL, DocOw11.5), 55,,,, and their state-of of-the-art performance and task generalization capabilities (LLaVa-NeXT) Phi-3.5).
Training Details. We conduct all experiments using nodes of 8 H100 GPUs, using Fully Sharded Data Parallel (FSDP) for distributed training. All experiments rate of 2e-5, with Adam W as the optimizer. More training details are provided in Appendix A.S.
batch size of 256 and learning
Evaluation Benchmarks & Metrics. In addition to the newly introduced BigDocs-Benc assess performance on well-known document-oriented benchmarks, termed as General Document Benchmarks. We select these benchmarks for their relevance and diverse range of document tasks. DOA (Mathew et al., 2021b), InfoVQA (Mathew et al., 2021a), DeepForm (Svetlichnaya, 2020), KLC (Stanislawek et al., 2021) (Kleister Benchmark for Key information extraction). WTQ (Pasupat & Liang, 2015) (Wikipedia Tables), TabFact (Chen et al., 2020), ChartQA (Masry et al., 2022), TeX (Singh et al., 2019), MMMU (Yue et al., 2024), DUDE (Landeghem et al., 2023), Slide- VQA (Tanaka et al., 2023), and TableVQA (Kim et al., 2024). We utilize (and extended) VLM Eval Kit (Duan et al., 2024).
In SigDoos-Benchh we employ the following evaluation methods based on each task's characteristics. For Screenshor2HTMM, inspired by related works in this domain (Reis et al., 2004), we compute Tree Edit Distance (TE Dist.) between the Document Object Model (DOM) of ground truth and generation. For Tab,e2LaTeX, we report TeXBLEU (Jung et al., 2024). For Image2SVG, we compare the cosine similarity between the DINOv2 (Oquab et al., 2023) representations of the ground-truth and generated SVG images (DINOScore). For the Image2Flow tasks, we propose the Length-Shape Triplet F1 (LST F1) score between ground-truth and generated flowcharts' edge sets. More specifically, each edge is represented as a (s, e, d) triplet, where e is the edge label, and & and f are the source and destination nodes' labels concatenated with their shapes, respectively. For Char12Markdown, we adopt the RMSF1 metric for markdown tables (Liu et al., 2022; Masry et al., 2023; 2024). For summarization and VQA tasks, we report Rouge-L F1 score (Lin, 2004). For more details about the evaluation process, please refer to Appendix A.6.
Setup. For our CPT on DocOw11.5-8B and Qwen2VL.-2B, we used base weights, while on Phi- 3.5-Vision-4B and Llava-NeXT-7B, we initialized from their instruction-tuned versions, since their base weights are not publicly available. We first trained each model on a CPT corpus, either DocStruct4M (Hu et al., 2024) or BigDocs-7.5M,, for one epoch. Following this, we performed further alignment (finetuning) using DocDownStream to enhance the model's ability to follow instructions. For each selected model, we also evaluated the provided base model and the instruction-tuned version (separate from the base checkpoint, if available) as basclines and reported the performance on general document benchmarks.
8

---

Published as a conference paper at ICLR 2025
Table Comparison of model performance in BegDocs-Bench. BigDoes models trained on (BigDocs- 7.5M+ DocDennstreetmmm BigDoss
train-splitt, which combine CPT and FT, ouperform all baselines in tasks requiring long-format code generation, particularly in flow generation. GUI reasoning, and image-to- LaTeX gemeration, surpassing even state-of- tatte-of-hhear the-art closed models. The symbol denotes that the model required I-shot prompts opposed to the default O- shor prompts.
Sererm CerrbbCHHTMN
Charzaa
Churtzt
Imoge ba
(Grophh Vin
ImagetFion
USeyNi
GULMaaton
Imegee DGVG
Knww
Table alex
GULVON
Nr Score
Model
Open Models
DedOw-1
13.60
0.00
0.001
13:88
3.58
15.32
Q1en2-VV-208
12:30
P-i3.5-V-48
60.64
21
1.51
27.80
10.11
34.57
4.25
24.14
34.96
22.13
11.VA-N:XX1-78
22.00
30.67
1.581
0.461
21.99
12:38
20.53
5.00
7381
27.54
20.60
Maliio2-BB
25.34
20:95
L.17
8.75
5,26
37.73
3.56
74.50
27.76
20.48
1.ma-922.906
45.21
20,00
0,75t
0:521
22.16
12.04
45.97
7.30
74.29
27.28
25.66
Qwent-V1-728
0.231
Chored Models
201408006
20
35.03
22
17:57
60.34
36.58
36.84
Chanke335 Scans
23.59
13:12
25.45
26,55
GcniniPro1155
76.63
25:90
11.511
33,597
25,54
16.79
15.21
7,43
75.22
35.35
32.32
BigfAres Models
DooOvlel 5-81s- BiglOcs
24.43
33.18
42,16
48.54
89.18
33.05
3.64
81.28
63.85
49.52
Qwent-Vi RipDees
23:25
71.65
33.51
9:20
78.54
33.97
46.92
I1NVA-NiXT--18- Rigftoos
23.76
99.66
79.55
68.63
Phi3 Moy3.eyyuBBB.. RigDous
84.01
36.78
71.86
47.32
36.91
34.65
44.81
56.34
We also provide a comprehensive evaluation on the proposed Big We conducted an off-the-shelf performance analysis on BigDocs-7.5M using models such as GPT4 (Achiam al., 2023), Claude (Anthropic, 2024), Gemini Pro (Team et al., 2024a) and Qwen2VL-72B (Wang et al., 2024a) and Idefics2 (Laurençon et al., 2024a). In addition, we also evaluated the previously selected models on their instruction versions, including DocOw11.5-88 and -e To incorporate the new capabilities introduced BigDocs- -Bnnch, we further finetuned these models after BigDocs CPT using the training set from BigDoss-Bench (see Table 3). For each model, we only evaluate the instruction-tunel version (where available) as baselines, reporting their respective performance.
Qwen2VL-225
-...35--ision-4B
5.2 QUANTITATIVE RESULTS
Results on Existing Document Downstream Tasks. Table 2 presents the models performance across general document benchmarks. Base models perform poorly, mainly due their inability to follow user instructions, like answering questions. Phi3.5 Vision, originally optimized for reasoning tasks, achieves average score of 60.80% when finetuned on BigDocs, enhancing its ability to handle complex document-based tasks. For Qwen2-VI. VI., an interesting pattern emerges: while the instruction-tune suneetion-tueedddeee version excels tasks reported in its technical report (e.g., Doc VQA, Info VQA, ChartQA, MMMU), the BigDocs-traines model surpasses it on new tasks like TabFact, DeepForm. KLC, and VQA. suggesting that Qwen2-VL's instruction tutiong may rely on complex prompt engineering and task task-specific optimizations. We find that performing additional continual pretraining and finetuning on instruction-
models does not significantly degrade performance. Moreover, we observe substantial improvements on previously underperforming tasks. As shown in Table 2, Phi3.5 and VLeee- show marked gains on DeepForm, KLC, WTQ, and SIi.eVQA. However, LLaVA's performance declined MMMU, indicating the need for more multiple-choice question data. This reinforces our argument that transparency in training datasets is essential for proper evaluation. Finally, across all models, BigDocs training yields higher average scores compared to training on DocStruct4M, even with lower contamination rates on most benchmarks, as we highlighted in Section 3. These findings indicate that BigDocs supports better generalization and robustness without complex task-specific optimizations while also being license- pennssspermisive..
Results on SeeDocs-Bench Tasks. Table 3 shows results for our proposed downstream tasks, evaluating models' ability to generate lengthy structured and valid code outputs, and reasoning from GUIS. Overall, BigDocs models consistently outperform both open and closed models on most tasks, particularly in Flow and GUI tasks such as GUI2UserIntent, GUI2Summary. GUI-VQA, and Image2Flow, revealing areas where existing models fall short. The performance gap is narrower tasks like Screenshot2HTML. Image2SVG and Char12Caption, suggesting these tasks have been explored in the literature but not extensively enough. Phi3.5-V4B BigDoes stands out the

---

Published as a conference paper at ICLR 2025
Table2LaTeX
Screetshoc2ETML
Phi BigDoos
DRAW 31S
DRAW
Ph:SS Highte WIN
WIN
GPT-4
BOTH BAID
Phi 3.5 BigDocs
BRAW
1,05
SigTack c9%
Bigkko
Phi 3.5 Instract
BOTHH BAI
BOTHE 1%
LOWS
Figure 5: Human evaluation results comparing Phi3.5 BigDocx-Benck Bench aguinst Phi3.5 Instruct and GPT-4o on tasks: Table2LaTex (Left) and Screensho(2HTMML (Right).
top performer in 8 tasks out of 10, with an average score of 50.46. However, its performance on Image2SVG (25.98 points behind I.LVV-NeXX-7BB BigDocs) indicates less exposure to SVG data in its instruction tuning. We find that the model can generate valid code outputs in different formats, including SVG, HTML, JSON, or Latex, when conditioned on images.
5.3 HUMAN EVALUATIONS AND QUALITATIVE RESULTS
We conducted human evaluation comparing the performance of Phi-BigDoes, Phi-Instruct, and GPT-40 on Screenshot2HTML and Table2LaTeX tasks. Twenty-eight evaluators participated, providing 1,900 annotations. For Table2LaTeX, evaluators assessed the LaTeX table matched the input table. For Screensho12HTML, they evaluated the visual similarity between the rendered HTML and the screenshot. Evaluators chose between four options: one of the two model outputs is superior to the alternative (recorded as "WIN" if it is Phi-BigDocs and "LOSS" otherwise), both outputs are acceptable but of very similar quality ("DRAW") and neither output acceptable ("BOTH BAD"). See Appendix A.8 for more details on the evaluation platform.
From human evaluation results in Figure 5, for the Table2 task, Phi3.5 BigDoes wins 88% of the time against Phi3.5 Instruct and achieves 63% win rate, with 31% draw rate, against GPT-4o. These results highlight our model's ability to accurately preserve the table's ' structure, including lines, borders, and margins, whereas GPT-40 often struggles maintain consistent formatting despite capturing content accurately. In the Screenshot2HTML task, Phi3.5 BigDocs achieves a 65% win rate against Phi3.5 Instruct and performs competitively against GPT-4o, with 36% win rate and 7% draw rate, demonstrating its strong capability to reproduce visual elements faithfully.
Qualitative Results. We provide qualitative results in Appendix A.10. Figure 8 presents outputs from experiments with the Phi-3.5-Vision model on BigDocs-Bende for tasks like Chart2Marddown, Table2 and Image2SVG. The model delivers visually consistent outputs and generates valid code across formats, adhering to task instructions. Table 6 compares sample outputs between Phi-3.5-Vision models and GPT-4o, highlighting the strong performance of our BigDocs trained version in captioning and VQA tasks.
CONCLUSION
We introduce BigDocs-7.5MM. large-scale, license- license-permisiiee permissive dataset for training multimodal models on document and code-related tasks. Along with a comprehensive suite of tools and data analysis, present BigDocs-Bence featuring 10 downstream tasks that assess model's ability generate long-format code outputs from images. These tasks serve as practical benchmarks for real-world applications. Our experiments show that models trained on BigDocs outperform those trained on existing datasets. Furthermore, training on the BigDocs-Beni with new capabilities and significantly enhances their ability to generate long, structured outputs. All BigDocs artifacts will be freely available under permissive licenses.
train split endows the resulting models
Limitations Our work presents a pioneering
dataset for multimodal document understanding, achieving strong performance across tasks. However, there are limitations: (I) Suboptimal performance on some public benchmarks, indicating a need to refine the data mixture and explore additional sources. (2) Limited context length, as models are trained with maximum of 8192 tokens, restricting performance on tasks with long, structured outputs like HTML and SVG. (3) Uncertainty in the commercial viability of base models, their pretraining data lacks transparency.
10

---

Published as a conference paper at ICLR 2025
ETHICS AND REPRODUCIBILITY STATEMENT
Ethics Statement Our work is centered around responsible and transparent curation of datasets for multimodal document understanding models. While we have made extensive efforts filter harmful content from our dataset, we cannot fully guarantee that the models will not generate offensive language. Additionally, we have taken significant steps to remove personally identifiable information (PII) from the compiled datasets to protect user privacy. However, we cannot ensure that the generated code will be free of malicious snippets, and developers are encouraged implement protection protocols to safeguard against potential risks. Finally, all human evaluation studies were conducted by collaborator researchers, and no PII was collected during this process.
Reproducibility Statement We committed to ensuring the reproducibility of our work by providing all necessary details and resources. All artifacts, including code, datasets, model weights, data sheets, and metadata, will be publicly released. Furthermore, we have fully documented all hyperparameters, experimental setups, and evaluation metrics to allow for accurate replication of our results. For human evaluation, we provide clear instructions and describe the environment used for comparison to ensure transparency and consistency.
ACKNOWLEDGMENTS
We thank Aishwarya Agarwal, Ghazwa Darwiche, Christian Hudon, Tom Murray, and Chao Wang for their precious administrative and/or technical assistance. Yoshua Bengio thanks Microsoft and Samsung for funding.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
Credit Mutuel Arkea. Aftdb dataset, 2024. URL https: htt.httt/h/hhgigggggeeee http:://hgggcccccco////oo twarkea/afftoe Hugging Face Dataset.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv: 2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vasion-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308, /2966, 2023b.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. In IEEE Conference Computer Vision Pattern Recognition, 2023.
Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz, 2024.
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mit- igating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587-604, 2018
11

---

Published as a conference paper at ICLR 2025
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang. and Percy Liang. The foundation model transparency index. arXiv preprint S.KV:23310.12941 2023.
Rishi Bommasani, Kevin Klyman, Sayash Kapoor, Shayne Longpre, Betty Xiong, Nestor Maslej, and Percy Liang. The foundation model transparency index v1.1. arXiv preprint arXiv:2310.12941, 2024.
Eukasz Borchmann. Notes on applicability of gpt-4 to document understanding. arXiv preprint arXiv:2405. 18433, 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conference Computer Vision Pattern Recognition, 2021.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. Leaf-qa: Locate, encode & attend for figure question answering. In Winter Conference on Applications of Computer Vision, 2019.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference Learning Representations, 2020.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.000
2015.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint MrXiv:2312.14238, 2023.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are we gpt-4v? closing the gap commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 2224044..821,, 2024.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li Yan, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Annual Meeting of the Association for Computational Linguistics, 2024.
Tri Dao. FlashAttention Faster attention with better parallelism and work partitioning. In International Conference Learning Representations, 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, et al. The llama herd of models. arXiv preprint arXiv:2407.2111 21733, 2024.
Wikimedia Foundation. Wikimedia downloads, 2024. URL https: //tpps//uupppppp///////// . wi k wikeiiiii... a org.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Gi- annis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander J. Ratner, Shuran Song. Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alexandros G. Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Adv. Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2025
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12) (1): 86-92, 2021.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: / / thub. www.cco/////oeeeeee hugg i in aace/ aaccelec erate, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning, 2022.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403. /2895, 2024.
Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: dataset for form understanding in noisy scanned documents. In International Conference Document Analysis and Recognition Workshop, 2019.
Kyudan Jung, Nam-Joon Kim, Hyongon Ryu, Sieun Hyeon, Seung jun Lee, and Hyeok jae Lec. Texbleu: Automatic metric for evaluate latex format. arXiv preprint arXiv: 2409,06639, 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visu- alizations via question answering. In IEEE Conference Computer Vision Pattern Recognition, 2018.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. In International Conference Learning Representations, 2017.
Yoonsik Kim, Moonbin Yim, and Ka Yeon Song. Tablevqa-bench: A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404. 19205, 2024.
kleister Charity. kleister charity dataset, 2021. URL https: com/ applicaai/ kleister-chhriis
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. Transactions Machine Learn Research, 2022.
Jordy Van Landeghem, Rubén Tito, Lukasz Borchmann, Michat Pietruszka, Pawel Józiak, Rafal Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Ackaert, Ernest Valveny, Matthew Blaschko, Sien Moens, and Tomasz Stanislawek. Document understanding dataset and evaluation (dude). In International Conference Computer Vision, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint S.Xiv:22005022440 2024a.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint wwXiv:2403.09022 2024b.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelies: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- laiiunl models? arXiv preprint arXiv: 2405.02246, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv: 0338003326 2024.
13

---

Published as a conference paper at ICLR 2025
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. Tablebank: A benchmark dataset for table detection and recognition. arXiv preprint arXiv:1903.0130 2020a.
Minghao Li. Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A benchmark dataset for document layout analysis. In International Conference on Computational Linguistics, 2020b.
Raymond Li, Loubna Ben Allal, Yangtian Zi. Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! Transactions Machine Learn Research, 2023.
Chin-Yes Lin. ROUGE: A package for automatic evaluation of summaries, In Text Summarization Branches Out. Association for Computational Linguistics, 2004.
Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by plot-to-table translation. arXiv preprint arXiv:22/2.10555, 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Adv. Neural Information Processing Systems, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https:// Alava-vi.gie wwg.hu..........g. ho/blog//022e2 01-00 30- alava-next/..
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang. Li Dong, Weiyao Luo, et al. Kosmos-2.5: A multimodal literate model. arXiv preprint ar arXiv:2309. 1419, 2023.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics. URL https: http://claaathhhoooooooo
org. 2022 acl short. 18.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. UniChart: A universal lasounnlanguage pretrained model for chart comprehension and reasoning. Empirical Methods in Natural Language Processing, 2023.
Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. ChartInstruct: Instruction tuning for chart comprehension and reasoning. In Annual Meeting of the Association for Computational Linguistics, 2024.
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny, and Jawahar. Infographicvqa. In Winter Conference Applications of Computer Vision, 2021a.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. In Winter Conference on Applications of Computer Vision, 2021b.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. International Conference on Document Analysis and Recognition, 2019.
OpenAl. GPT-VVissoo) System Card. https:// tttps://cdaa wttpa/...pp......oooo wpppenaa.co/pppppe///ee System_Carde pdf, September 2023, Accessed: 2023-11-05.
14

---

Published as a conference paper at ICLR 2025
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199, 2023.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. In Adv. Neural Information Processing Systems, 2011.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: A consolidated receipt dataset for post-ocr parsing. In Adv. Neural Information Processing Systems Workshop, 2019.
Panupong Pasupat and Percy Liang. Compositional semantic parsing semi-structured tables. In Annual Meeting of the Association for Computational Linguistics, 2015.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, 2021.
N Reimers. Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 2019.
D. C. Reis, P. B. Golgher, A. S. Silva, and A. F. Laender. Automatic web news extraction using tree edit distance. In International Conference on World Wide Web, 2004.
Juan A Rodriguez, Shubham Agarwal, Issam Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, and Marco Pedersoli. Starvector: Generating scalable vector graphics code from images. arXiv preprint arXiv:2312. 2023a.
Juan A Rodriguez, David Vazquez. Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Figgen: Text to scientific figure generation. In International Conference Learning Representations, 2023b.
Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Ocr-vggan: Taming text-within-image generation. In Winter Conference Applications of Computer Vision, 2023c.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: dataset for image captioningwith reading comprehension. In European Conference Computer Vision, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In IEEE Conference Computer Vision Pattern Recognition, 2019.
Amanpreet Singh, Niranjan Balasubramanian, and A B. Open4business(o4b); An open access dataset for summarizing business documents. arXiv preprint arXiv:2011.07636 2020.
Amanpreet Singh, Guan Pang. Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Textocr: Towards lcrge-scale end-to-end reasoning for arbitrary-shaper scene text. In JEEE Conference Computer Vision Pattern Recognition, 2021.
15

---

Published as a conference paper at ICLR 2025
Brandon Smock, Rohith Pesala, and Robin Abraham. Pubtables-Im: Im: Towards comprehensive table extraction from unstructured documents. In IEEE Conference Computer Vision Pattern Recognition, 2022.
Ian Soboro. Complex document information processing (cdip) dataset. https://dbi.org// 18434/mas 2/d44/mm22---0 2531, 2022. Accessed: 2024-06-20.
Tomasz Stanistawek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemyslaw Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In International Conference on Document Analysis and Recognition, 2021.
S Svetlichnaya. Deepform:
Understand structured documents at scale, 2020.
Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering multiple images. In Association for the Advancement of Artificial Intelligence, 2023.
Simon Tannert, Marcelo G. Feighelstein, Jasmina Bogojeska, Joseph Shtok, Assaf Arbelle, Peter W. J. Staar, Anika Schumann, Jonas Kuhn, and Leonid Karlinsky. FlowchartQA: 'he first large-scale benchmark for reasoning over flowcharts. In Piush Aggarwal, VOOzge Alalccam, Carina Silberer, Sina Zarrie'ss, and Torsten Zesch (eds.), Proceedings of the 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, pp. 34-46, Ingolstadt, Germany, September 2023. Association for Computational Lingustics. URL https: /tap:s://cclaas http://aaaaaanooo....o.a. org/2023. 1imo-1.5.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, and Others. Gemini: A family of highly capable multimodal models. 2024a.
arXiv preprint arXiv:2312.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding. Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, and Others. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0555 2024b.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang. Adithya Iyer, Xichen Pan, al. Cambrian-1: A fully open, vision-centric exploration of multimodal lims. arXiv preprint arXiv:2406. 16860, 2024.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv: 1601.07 1.01007140 2016.
Caitlin Vogus and Emma Llansóe. Making transparency meaningful: A framework for policymakers. Center for Democracy and Technology, 2021.
Stěpán Šimsa, Milan Šule, Michal Uřičár, Yash Patel, Ahmed Hamdi, Matëj Kocián, Matyáš Skalicky, Jifí Matas, Antoine Doucet, Mickaël Coustaty, and Dimosthenis Karatzas. Docile benchmark for document information localization and extraction. In International Conference Document Analysis and Recognition, 2023.
Bryan Wang. Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In ACM Symposium on User Interface Software and Technology, 2021.
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Arminch Nourbakhsh, and Xiaomo Liu. Docllm: A layout-aware generative language model for multimodal document understanding. arXiv preprint arXiv: 2401.00908 2023a.
16

---

Published as a conference paper at ICLR 2025
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- langugge model's ' perception of the world at any resolution. arXiv preprint 2xX1v2240.121111, 2024a.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024b.
Zilong Wang, Yichao Zhou, Wei Wei, Chen- Yu Lee, and Sandeep Tata. Vrdu: A benchmark for visually-rich document understanding. In International Conference on Knowledge Díscovery & Data Mining, 2023b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint
2rX.v:2309.17422 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In IEEE Conference Computer Vision Pattern Recognition, 2024.
Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, and Yoshua Bengio. Ver: Visual caption restoration. arXiv preprint arXiv: 2406.06462. 2024.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.1 11277, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint 24Xiv:241.001144 2024.
Fengbin Zhu, Wenqiang Lei, Fuli Feng, Chao Wang, Haozhou Zhang, and Tat-Seng Chua. Towards complex document understanding by discrete reasoning. In ACM International Conference on Multimedia, 2022.
A APPENDIX
A.1 BIGDOCS-7.5M TASKS
1. OCR-Related Tasks:
These tasks involve converting images of text (e.g., scanned docu- ments) into machine-readable formats, including transformations such bounding-box-to- text and eext-to-bounding-box (text localization as described in Hu et al. (2024)). Models learn recognize and map textual information within images.
2. Structured Parsing and Extraction: This task focuses on extracting and transforming structured data from documents, like parsing tables, forms, and charts into formats such as JSON or Markdown. It includes handling documents with complex visual layouts and sometimes incorporates bounding box information for individual elements.
3. Captioning and Summarization:
This task requires models to generate captions sum- maries for visual or textual content, such as figures, charts, or document sections. The models provide concise descriptions or overviews, enhancing document comprehension.
4. Question-Answenr Answering (QA): QA tasks involve responding to questions posed over structured or visual data (e.g., tables, figures). This includes multi-turn QA, where models address a series of related questions improve their reasoning and comprehension.
17

---

Published as a conference paper at ICLR 2025
A.2 DATASETS INCLUDED IN BIGDoCS-7.5M
The following datasets are utilized in our work, supporting the four core tasks mentioned in Sec- tion 3.2.1. We reference the task supported by each dataset via the index in Section A.I. Note that datasets with are those fully or partially curated by us (details explained in the description of each of them).
1. TabFact (Chen et al., 2020): [Task included: (2), (4) TabFact is used for question-answerriee tasks, where models check whether a given statement is supported or refuted by a table. It also involves structured parsing, helping models extract and process structured table data into formats like Markdown.
2. Open4Business (O4B) (Singh et al., 2020): [Task included: (2), (3)] O4B is dataset focused on business-related documents and is processed for structured parsing and extraction, as well as captioning and summarization, allowing models to retrieve key insights from documents and generate summaries or descriptions.
3. WikiTQ (Pasupat & Liang, 2015): [Task included: (2), (4) WikiTableQuestions is used for question-answwriig tasks over tables, where models answer questions based on table data, and structured parsing for converting table data into Markdown.
4. CORD (Park et al., 2019): [Task included: (1), (2)) CORD a dataset for parsing receipts, used for both OCR-related tasks and structured parsing and extraction, helping models interpret structured financial data from document layouts. This latter task requires extracting entities and providing their text, category, and location JSON output.
5. UniChart (Masry et al., 2023): [Task included: (2), (4)) UniChart is used for structured parsing and extraction to extract structured information from chart-like tables, converting complex visual layouts into formats like JSON Markdown. And also for question- answering tasks, where models answer questions related to the chart content.
6. TextOCR (Sidorov et al., 2020): [Task included: (1)) Tex1CCR is processed for OCR-related tasks, enabling models to perform bounding boree-booottteeeeee transformations scene text from images.
7. COCO-Text (Veit et al., 2016): [Task included: (1) COCO- used for OCR-related tasks, helping models extract and recognize text from real- cal-world images with natural scene text.
8. CDIP-1M": [Task included: (1), (2)) To create this dataset, we sourced from the original IIT-CDIP (Soboro, 2022) dataset and curated it into CDIP-1M. CDIP-1M is processed for OCR-related tasks and structured parsing, focusing on extracting text and structure from large-scale scanned document collections. We used an in-house OCR engine to get the text (i.c. annotations) from its 1 I1M documents from the source. Like the text localization task in DocStruct4M Hu al. (2024), we generate word-, line- and block-level bounding-boo box-too text and eoxtto-bboudding box QA pairs. We subsample zones randomly, and based on OCR confidence, large fraction of the images are pretty noisy. In addition, for the block level, we generate structured parsing QA pairs where text lines and their location need to be given as JSON by the model.
9. Pub Tablee----MM 1M (Smock et al., 2022): [Task included: (1), (2)) PubTablee- MM a large dataset of tables from scientific papers. It is used for OCR-related tasks to extract informa- tion in tables. It is also processed for structured parsing and extraction, allowing models to handle scientific tables and convert them into markdown.
10. FigureQA (Kahou et al., 2017): [Task included: (4)] FigureQA is focused on question- answering tasks, where models answer questions based on charts and figures, improving reasoning over visual and tabular data.
11. DocBank (Li et al., 2020b): [Task included: (2), (4)) DocBank is utilized for structured parsing and extraction and question-answente answering tasks, enabling models to interact with scholarly documents and extract structured data from layouts.
12. TableBank (Li et al., 2020a): [Task included: (2)) TableBank is large dataset used for structured parsing and extraction, helping models parse table structures from both Word and LaTeX documents into structured formats.
18

---

Published as a conference paper at ICLR 2025
13. OCRVQA (Mishra al., 2019): [Task included: (1), OCRVQA focuses on OCR-related tasks and cresston-answeriigg tasks over OCR-extracted text, where models answer questions based on text and visual data from real-world scenes.
14. Datikz (Belouadi et al., 2023): [Task included: (2), (3)) Datikz is processed for captioning and summarization as well as structured parsing, enabling models describe and interpret complex diagrams and generate structured data from them.
15. ArxivOCR': [Task included: (1)) The ArxivOCR dataset contains OCR-scamen academic papers and used for OCR-related tasks, where models perform bounding-box -aoegg--boto----eooo -tx-texxt transformations, improving accessibility and structure for scholarly articles. This dataset is curated by us. We filter out the papers from Arxiv that have permissive licenses, i.c. CC-BY 4.0 and CCO in this case. Then we use in-house OCR engines to produce OCR results on the pages from the papers collected.
16. Arxiv VexivTableCap'. [Task included: (3)] The ArxivTable dataset focuses on generating captions for the tables and figures extracted from Arxiv papers, helping models describe the content and context of tables in academic papers. Among the 156.2k samples, 70k of them are from AFTdb Arkea (2024). We perform the filtering to make sure all the selected ones are in papers with permissive licenses, ie.. CC-BY 4.0 and CCO in this case.
17. SVGCap Dataset (Rodriguez et al., 2023a): [Task included: (3)) The SVG Dataset supports captioning and summarization tasks, where models generate descriptive captions for SVG content, summarizing the structure and elements of vector graphics.
A.3 ASSESSING CONTAMINATION
BigDocs- 775M''s direct dependency TabFact, WTQ, and TextVQA is restricted to their training splits, and DocStruct4M reports similar dependency on Voc Info DeepForm, KLC, ChartQA, TabFact, WTQ, and TextVQA. However, overlaps between either of these training sets and evaluation datasets may emerge through indirect means (c.g., the same source material was involved).
Our primary "automatic" approach to assess contamination consists of embedding all images from the reference dataset (i.e., BigDocs-7.5M or DocStruct4M) using a pretrained CLIP (Radford et al., 2021), namely -ii-Vii-B--- 32 from sentence- sereeceeeennssseee raans formers (Reimers, 2019). We similarly embed each image from an evaluation dataset and retrieve the reference image with the highest cosine similarity: the closer this measure to 1.0, the more likely it that the evaluation image is part of the reference dataset. Figures 3 and 6, as well as table 4 all report these values.
Except when the cosine similarity is exactly 1.0 indicating an exact match interpretations of these scores must be grounded in human evaluations. However, assessing whether two images are "the same" can be a non-trivial task, even for human eyes.
Consider the following edge cases:
receipts for recurring orders emitted on different days;
the same form filed by different people;
different versions of the same form at the same company; and
different full-page table from the same appendix of the same report.
Technically, all these cases involve pairs of different documents. However, one could argue that training on one such samples confers an unfair advantage to a model evaluated on the other sample. For this reason, we annotate such instances as same template whenever we encounter them.
Conversely, the "actual" same document can appear as quite diversified images. Indeed, stamps, watermarks, censorship, and/or annotations may be apposed posteriori to document, in addition to scaling, crops, rotations, and fax-induced artifacts. In all these cases, the original intent to communicate the same information matters: different copies of the same memorandum, scanned separately, are here treated as the same document.
With these definitions in hand, a human is tasked with labeling the samples that most likely to be problematic in the Doc DocVQA evaluation dataset. Figure 7 reports the results, calibrating how we interpret the cosine similarities for other evaluation datasets.
19

---

Published as a conference paper at ICLR 2025
109
W.H. 7MM5
w.F.r DoeStrmei4M
DoeVQA
DocVQA
Proportion shove threshokd
10"
InAQQA
InfoVQA
DespForm
DeepForm
K.C.
KI.
WTQ
WTQ
10
TabFact
Chariq
CharQA
TexiVQA
TeaVQA
MMMU
MMMU
10
DudeMini
DideMini
SlideVQA-M
SULVVA--M
TableVQA
TablVQQ
0.60
0.65
0.70
0.75
0.50
0.85
0,90
0.95
1.00
Threshold vosine similarity (embodded with clip- VII-B-32)
VQA. SladeVQA-M, and Fact overlop very little with BigDocs-7. (mrain).
10
Proportion above threshold
10"
0.92
0.93
0.94
0.95
3.96
0.97
0.98
0.99
Threshold cosine similarity (embedded with clip- -iT--VI
(b) Zoom the above (same legend).
shows the proportion of samples an evaluation dataset for which there exists least one sample in reference dataset with cosine similarity higher than the specified threshold. Lower values are better; higher cosine similarity thresholds are more pertinent. Based on human evaluations, samples with cosine similarity < 0.96 are unlikely be problematic, whereas those 0.98 are most likely problematic. Except for MMMU, and DudeMini, BigDocs-7.o appears less contaminated than DocStruct4M.
Figure 6: Cumulative distribution by cosine similarity. Each
Table 4: Detailed Results on Contamination Experiments, related to Figure Lower values are better; higher cosine similarity thresholds are more pertinent. Except for MMMU and DudeMini, BigDocs-7755 17.s-7..5M appears to be less contaminated by these metrics than DocStruct4M, Figure 3's data comes from this table.
SRdeVAA-M
Ceshte Sbalarity Throsbald
Docy QA
InfoVOA
DempForm
TahFari
Charlis
TodVQA
MMMU
DudiMMni
TobleVOA
BLC
WTQ
Reference Dutuset
0.99
DocSmruc4M
0.016
0.0
0.037
0.0033
0.00039
0.034
0.0
0.0026
0.0089
0.0027
BigDacx-7.5M
0.98
DocAmmeráM
0.065
0.0032
0.20
0.061
B.006
0.36
0.0020
000044
0.013
0.006
BigDoes SM
0.012
0.0
0.9
0.1033
0.0
0.0
0.10
0.0
0.0033
0
0.0
0.0053
D
BigDoes-7.SM
0.049
0.0
0.0
0.043
0.0
0.17
0.0
0.0061
0.053
0.0053
0.96
DocSmmcct4
0.26
0.0557
0.57
0.24
0431
0.14
0.63
0.0966
0.9470
0.15
0.0014
0.017
BigDoex-7.5M
0,0087
0.95
DocSmuetdM
0.39
0.057
0.72
0.30
0.62
0.20
0.68
0.012
0.015
0.28
0.0014
0.04
HigDoes-7.5M 7.5
0.24
(.01
0.21
0.23
0.0
0.0
0.45
0.01
0.045
0.22
0.0012
0.023
Number samples
5149
2901
1500
4872
4543
12722
2500
5000
1140
5275
196000
1.00)
Number unique
1281
500
300
608
421
1693
1509
3166
1100
609
3596
751
20

---

Published as a conference paper at ICLR 2025
1st
2nd
Unlabeled
133
19
79
72
0
13
12
0
0
Different
3rd
Same template
132
13
55
83
3
65
00
12
0
0
0
Same document
0,92
0.93
0.95
0.96
0.97
0.98
0.99
clip-
B-1-BB32 cosine similarity
With respect Bigs top samples haman-labeledd
1st-
172
146
100
0
11
2nd
Unlabeled
140
117
11
25
0
0
Different
3rd
Same template
155 11
132
83
9
14
20
21
12
0
Same document
0.92
0.93
0.95
0.96
0,97
0.98
0.99
1.00
vii--Vit-B-32 cosine similarity
(b) With respect DocStruct4;; top 100 samples humon-labeled
Figure 7: Human evaluation of Doc VQA's overlap. Among the 1284 unique images in DocVQA's samples, the same cosine similarity method figure identify the samples that the most similar samples corresponding training set. Although we prioritize using only closest match, we also retrieve the second and third closest matches after deduplication (i.e.. ... if the next closest match identicul to the previous match, skip it). A human is then tasked to label the top matches as either "different" "same template" or "sume document" " (see tex for definitions). Counts are provided for the most relevant 0.01-wide cosine similarity intervals. Most samples below 0.96 are "different", and most samples above 0.98 are not, From these numbers, we expect less than 10% of the samples with cosine similarity below 0.97 to be "same template" ", and less than 1% of the samples with cosine similarity below the same threshold to be "same document" All identified "same document" are found at the first rank. There is only one instance where the first rank is "different" but "same template" is identified at higher rank.
Note that this labeling procedure does not alter the composition of BigDocs-7.5M: we leave all samples, problematic or not, in the dataset. Instead, we release the annotations, which may help the community develop an intuition of the overlaps that may not have been identified yet, or even enable better detection strategies in the future.
A.4 DESCRIPTION OF DOWNSTREAM TASKS PROPOSED IN BIGDOCS-BENT
The following is formal description of the downstream tasks aim to solve using the proposed BigDocs-Bent dataset.
A.4.1 IMAGE2FLOW
The task at hand is an image-to- flow conversion, where the input is an image of a flowchart, and the output is the corresponding information in JSON format. This JSON includes the nodes and edges that represent the flowchart's structure.
Formally, given image I representing a flowchart, the goal is to extract graph G (V, E) where V is the set of nodes, and the set of directed edges between these nodes. The output JSON(G) contains two main components: (1) list of nodes V with their corresponding attributes such node
21

---

Published as a conference paper at ICLR 2025
autte KoOM
12.48
Phi 3.5 Vision BigDoes
XCF
Transkers Bh perrsiic-chtr
PIONI gEncTIC SVG dVENE
Figure 8: Qualitative Results. Generations of our Phi3.5 Phi3.S-Vision model on the presented document downstream tasks, part of the test set of BigDocs Bench. We show samples Char12Mrrkdown, Table2LaTex, Image2SVG. and lmage2Flow (JSON). single model trained BigDocs datasets can generate image-conditioned code different coding languages while providing valid outputs.
ID, label, shape, and description, and (2) list of edges E with their source node ID, destination node ID, and label (if any).
The dataset for this task consists of 10,000 samples. Each sample includes an image of flowchart, its corresponding Graphviz file, and a JSON representation.
The dataset was generated using the following pipeline: First, random number between 5 and 15 was generated to determine the number of nodes in each flowchart. Based on the total number of nodes, distribution was assigned to different types of nodes, such conditions (diamond shape), statements (rectangular nodes), and others (parallelogram, circle). A random flow direction was then selected from the options: BT (bottom to top), TB (top to bottom), LR (left to right), and RL (right to left). Using the LLaMA 3.1 model, we used the prompt in Table A.4.1 to generate Graphviz file.
Prompt used with Llama 3.1 to generate Graphviz data
Create directe flowchart graph in DOT format with the following specifications
The direction of the graph should be (direction).
Total number of nodes: (total_nodes). Nodes distribution: (nodes).
The edges should connect the nodes in a way that makes sense: Each node should have at least one outgoing edge.
For adiamond' nodes, there should be at least two outgoing edges. The graph can include any colors or additional styling. Generate valid and coherent graph. The graph should represent an enterprise-level workflow like the steps of an incident resolution workflow. The enterprise-level workflow labels are so important. Just give me the graph in Graphviz format no need for any python code any description about the graph.
Random colors and styles were applied to remove model biases toward specific stylings. The generated Graphviz files (.gv) were then converted to PNG images and JSON files. The JSON files contain two main components: Each node has ID, label, shape, and description of the shape (e.g.. diamonds represent conditions). Each edge contains the source and destination node IDs and label if present (the label represents the text on the edge).
22

---

Published as a conference paper at ICLR 2025
A.4.2 SCREENSHOT2HTML
The Screenshot-tor toaishoottt--TMLL conversion task involves transforming an image of a website's 's layout, such screenshot, into the corresponding HTML code that accurately reconstructs the structure and content of the original website. This process enables the generation of a functional website solely from its visual representation, facilitating applications in web design automation, accessibility enhancements, and rapid prototyping.
Formally, given an input image depicting the visual layout of a website, the objective generate the HTML structure H/ that includes essential web elements such as headers, paragraphs, images, links, forms, and navigation bars. The resulting HTML code HTML(H) should not only replicate the visual appearance of the original website but also ensure semantic correctness and structural integrity, enabling the recreated website to be interactive and accessible.
Data Collection and Filtering Process. The Image-to-HTML. dataset was curated through an automated pipeline designed to ensure the quality and diversity of website layouts. Utilizing the Playwright library, the system asynchronously retrieves and renders websites from compre- hensive list of URLs (Penedo et al., 2024). Each website undergoes a series of checks to verify its accessibilityy compliance with robots txt directives, and predominance of English content. Additionally, content appropriateness is assessed by filtering out websites containing NSFW language.
External CSS and JavaScript resources are inlined to maintain consistency and reduce dependencies, and unnecessary or oversized seripts are removed. Images are replaced with placeholders, and background images are eliminated focus structural elements. The viewport is adjusted to capture only the visible portion of cach page, enhancing the clarity of the layout representations.
Websites are further evaluated based on performance and structural metrics, including load time, page size, number of JavaScript and CSS files, DOM depth, and total number of HTML elements. Technologies and frameworks used by each website are identified to exclude those utilizing disallowed technologies, e.g., allowing a website to render without JavaScript. We also removed comments and prettified the HTML for standardization. Only websites that meet all predefined criteria are included in the final dataset.
Dataset Statistics. The resulting dataset comprises 11,000 website samples, each with a high- resolution screenshot and its corresponding HTML representation. On average, each layout contains 20.3 HTML elements, reflecting a diverse range of website designs. This diversity provides a robust foundation for training and evaluating image-to-HTML conversion models, ensuring the dataset is both comprehensive and representative of various web structures.
A.4.3 TABLE2LATEX
The task involves identifying and associating tables in academic PDFs with their corresponding LaTeX code and captions. The aim to precisely match each table image with the LaTeX source used to generate it and the relevant caption. ensuring accurate alignment between the visual content and its textual description.
More specifically, given table image 1, the LaTeX code C used to render the table, and the caption Teuption: the goal is to create a dataset Dataset C, Tespsion) that establishes reliable association between the tables, their LaTeX source, and their descriptive captions.
We crawled publicly available, license-compp complian arXiv papers to create this dataset, collecting both their PDFs and associated TeX source files. We began by parsing the TeX files to extract the LaTeX code for tables and their captions. Next, we used the PDF parsing library PyMuPDF to locate tables within the PDFs. However, this table detection process proved to be imperfect, as the algorithm frequently misidentified content with parallel lines as tables, leading to false positives.
To address this challenge, we adopted alternative approach. Instead of cropping tables directly from the PDFs where false positives were common- we chose to render the parsed LaTeX table code to generate accurate table images. This method ensured that the images faithfully represented the original table formatting, reducing detection errors and improving the reliability of the dataset. As result, created high-quality dataset comprising over 95,000 table images, each paired with
23

---

Published as a conference paper at ICLR 2025
its corresponding LaTeX code and caption, providing valuable resource for further research into table structures in academic papers.
A.4.4 IMAGE2SVG & TEXT2SVG
Scalable Vector Graphies (SVG) provide precise alternative to pixel-based images, capable of representing diagrams, icons, plots, and graphic designs with superior detail and scalability. Unlike raster images, SVGs can be scaled to any resolution without losing quality. In this work, we introduce the task of image-to-SVG generation, where the goal is to process an input image and generate SVG code that closely resembles the image upon rendering. This task requires advanced parsing capabilities for textual and numerical data and an understanding of various shapes, such as squares and arrows, commonly found in diagrams. Given input image I, the model outputs SVG code C' that visually replicates the image. Additionally, the task can extend to scenarios where a textual description T serves input, yielding SVG code that aligns with the described content.
We curate a dataset of images, SVG codes, and texts sourced from the SVG-Stack dataset introduced by StarVector (Rodriguez al., 2023a). The curation process involves filtering to ensure high-quality samples. First, we filter based on image entropy to select images with complex designs and intricate details, excluding simpler icons or shapes. Second, we compute the CLIP Score (Hessel et al., 2022; Radford et al., 2019) for image-text pairs and retain the top 100k examples to build our curated SVG dataset. The SVG-Stack dataset adheres to permissive licensing standards, originating from TheStack ( (Kocetkov et al., 2022), carefully designed for open and permissive use.
A.4.5 CHART2MARKDOWN
This dataset is novel contribution of our work, designed assess the models' capabilities in extracting data values from chart images. In this task, the model is given a chart image 7 and asked to produce data table T of the underlying data table markdown format.
To create this dataset, we crawled recent chart images from the Statista website ", focusing on charts from 2023 and 2024 that were not used in prior datasets like UniChart Masry al. (2023) and ChartQA Masry et al. (2022). This ensures that the dataset reflects the most up-to-date facts and trends and overlaps less with existing datasets and benchmarks. We collected 6,516 chart images, their corresponding data tables, and numan-written summaries.
A.4.6 CHART2CAPTION
The task at hand is chart-to-caption conversion. The input consists of image of chart along with the code used to generate it and the dataset's 's name and description. The output is a textual caption of the important insights and information conveyed by the chart.
Formally, given a chart image I, the code C used to generate the chart, and the dataset information D, the goal to produce a caption Caption C, D) that highlights key insights and information represented in the chart.
The dataset for this task consists of 1,496 pairs of chart images and the corresponding code that generated these charts. The data was collected by selecting various Kaggle public datasets and their associated data analytics notebooks. We executed these notebooks locally and parsed their outputs to generate the chart-image and code pairs.
To generate the captions, we used the prompt below with the provided chart image, code, dataset name, and description. The caption was generated by using InterVI.2-2661 (Chen et al., 2023; 2024). This process allows us to leverage the model's capabilities to generate meaningful captions based on the provided context of the chart, code, and dataset description.
Prompt used with VnternVL2-22 VL2-26B to generate chart summaries
You powerful data analyst. This is notebook with name " "[dataset_namel In the description of this dataset, it's
hhtpss//wwwwstaaistn.comm
24

---

Published as a conference paper at ICLR 2025
told that: "(dataset ietraassttdescrppiiiin. You are seeing a plot
from this notebook. Here is the code that was used to generate this plot:
(code)
Now as a data analyst, summarize the important insights and information about the chart.
A.4.7 GUI2USERINTENT
The GUI2UserIntent task tests the abilitics grounding on GUI. Concretely, given the bounding box coordinate clicked by the user, the goal is to identify the text element the intends to interact with. While this is similar to the GUI grounding task introduced in Cheng et al. (2024), which predicts the bounding box based are curated by repurposing the pretraining dataset for SeeClick (Cheng et al., 2024). The original dataset includes clickable text elements and their bounding box coordinates in website screenshots. We directly extracted them to curate the GUI2UserIntent dataset.
user query, it differs in its focus on interpreting user clicks. The datasets
A.4.8 GUI2SUMMARY
The GUI2Summary task is similar to the Chart-to-Summary Summary task in NovelCharts; however, instead of charts, the input images I are website screenshots. Unlike existing UI summary datasets like Screen2Words (Wang et al., 2021) that focus short, phrase-level summaries. dataset provides paragraph-level summaries. These summaries are comprehensive, providing a brief overview of the main content, referencing key visual and textual elements in the screenshots, and additional aspects like layout and color schemes. To synthesize data for the GUI-to-Summan task, we used Intern VL2-8B (Chen et al., 2023), prompting it with website screenshots to create concise descriptions of the main content and layout. Appendix A.4.8 shows the specific prompt used for data generation.
GUI-to-Sumna
Prompt used with Intern VT.2-8B to generate website sereenshot summaries
Summarize this website less than 100 words. Cover the main content, layout, color, and other style elements.
A.4.9 GUI-VQA
In the GUI-VQA task, the model answers questions about website screenshots, covering the overall content and specific elements like buttons, text boxes, and menu items. This task requires recognizing key components within the GUI and understanding their functionalities and interdependencies. We synthesized data for GUI-VQA using the GU12Summary dataset. Specifically, we sampled a sentence from each summary and prompted L.LMMA 3.1 -b1-8b (Dubey et al., 2024) to convert it into QA pair. The prompt is shown Appendix A.4.9.
Prompt used with Intern LL2-8B to convert summary to QA pair.
You will be provided with a sentence. Your task is to convert it into question-answa pair. The question should focus on factual information and avoid subjective inquiries. Do not generate Yes/No questions. Structure the response in the format:
Q: (question)
A: (answer)
A.5 DETAILS ON TRAINING VISION-LANGUAGE MODELS ON BIGDOCS
Figure 11 illustrates our training pipeline for evaluating the quality of BigDoes and its impact compared to other datasets. The process has three stages: (1) Pretraining, which focuses general
25

---

Published as a conference paper at ICLR 2025
GUI-to-Sune STT-to--Summryy
GUI-VQA
OBC
cficcion
Input InaessQuueries
What does the online platform primartly deel with?
This website for "Casa Giordano, restaurant spectalizing Italinn cutmee content dish calied 'Arangin Trufic," priced 64.00. The modern, red
top navigution "Aecuell" "Click "Lvénements"
Output Text
"Contact. Icons shopping and finvorites design minimalistic, with a focus food image clear pricing information.
Inftueza vaccination and its related information
Figure 9: GUI-to-Summan -Suumary and GUI-VQA introduced in BigDocs-Bencee
2022
2001
79.17
2010
06.10
2015
2016
2012
2013
$3.70
2011
2016
Figure 10: Chart-to-Mrrddow task introduced in BigDocs-Bench.
26

---

Published as a conference paper at ICLR 2025
Stage 1: Visiot-Languuge Pre Trnining (PT)
Stage 2: Continoal Fre Training (ePT)
Stage Supeervised-innettrree (SFT)
General Vision-Lorgunge Alignment
Infaning Dacument-moge Understonding
Prepare for Downetreom Document Zooks
Image Phooder
Image Eucoder
Image Encoder
Language Modaling Objecttive
Tauguage Modeling Objective
Language Modellng
Cocnectirr
Connector
Connector
Objjcctive
Large Language Model
Large Language Model
Large Language Model
Training Data and Tasks
Training Data and Tasks (BigDooss) Documeents, websine, text intensive Parsing, Captioning, Code Generation, OCR
Training Data and Tasks (RigDect-FI) Documents, wehstte, inteasivv Instruction Followeig VQ4, Generation
Web- Scale image natural Image Captioning, Mulli-Turn Conversation
Figure 11: Training Stages for BigDocs. Three-Stage Training Pipeline for multimodal Document Understand- ing with BigDocs. Our approach consists of: (1) General Vision-Languee Pretraining, (2) Document Speciffcc Continual Pretraining, and (3) Supervised Finetuning for Document Tasks. We evaluate the impact of BigDocs by using checkpoints of models after Stage 1, as provided by their original authors, and comparing them with models that undergo all three stages. Performance assessed based on standard document tasks and novel tasks, including HTML/SVG code generation, flowchart generation/parsing,, and LaTeX interpretation.
modality alignment; (2) Continual pretraining, where models are further aligned to specific domain, such as documents, using general tasks like OCR and captioning; and (3) Finetuning, which uses smaller datasets to prepare models for specific downstream tasks, like document processing.
We did not perform stage in this paper, instead relying on checkpoints after pretraining. Additionally, we conducted some experiments using instruction-tuned checkpoints. Throughout all three stages, we employ generative loss objective, specifically a categorical cross-entropy loss, to predict the next token given the context. The goal of BigDocs-7.5M is to enhance the model's understanding and alignment with document-specific structures. At the same time, BigDocs-Benee focuses on training models to handle tasks that require processing images and converting them into structured code representations. These outputs often follow strict validity constraints to ensure they are syntactically correct. Additionally, the BigDocs-Bench test sets introduce novel benchmarks comprising five company-related downstream tasks, further validating the model's 's ability to generalize to real-world document-based applications.
Training Details. We conduct all our experiments on a cluster of 64 H100 S0GB GPUs leveraging the Transformers library (Wolf et al., 2019) for model development. Using Accelerate (Gugger et al., 2022) and Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023), we distribute training over all GPUs in our cluster. We wrap the transformer blocks in both the encoder and the decoder into separate FSDP units with activation checkpointing to achieve a data parallel of 64 without rumning out of memory. FlashAttention-22 (Dao, 2024) gives a significant speedup on our training runs. For reproducibility. we provide the hyperparameters and training details of our experiments. All experiments maintain a constant total batch size of 256 and a learning rate of 2e-5. We utilize the Adam optimizer with the following parameters: a cosine learning rate scheduler, 60 warm-up steps, a betal coefficient of 0.95, a beta2 coefficient of 0.999, a weight decay of le-6, and an epsilon value of 1e-8. For the CPT experiments, models are trained for 1 epoch, while the finetuning experiments on the DocDownstream dataset and BigDocs- DigDocs-Benchh are conducted over epochs.
A.6 DETAILS ON MEADOCS-BENCHT ' EVALUATION
In this section, we dive into more details of the evaluation process for SigDocs-Benchh including the preprocessing procedure before the evaluation and the implementation details of the metrics,
A.6.1 PREPROCESSING
Before conducting the evaluation, we perform the following preprocessing step for both the ground truth and generation texts:
1. Image2Flow (Graph Viz): the text contains markdown-style fenced code blocks, we use regular expression extract the contents within the first code block; otherwise, we treat
27

---

Published as a conference paper at ICLR 2025
the entire text Graph Viz code and remove anything before digraph or graph if either exists. The flowchart triplets are extracted with the pydot library.
2. Image2Flow (JSON): If the text contains markdown-stylk fenced code blocks, we use regular expression to extract the contents within the first code block; otherwise, we remove all contents outside the first "(" and last ")" the text.
3. ScreenshotHHTMI: If the text contains markdown-style fenced code blocks, we use a regular expression to extract the contents within the first code block; otherwise, we treat the entire text as the output HTML code. We first normalize the HTML code with htmlmin minify () which removes the comments in HTML and condenses the at- tributes their most miniature possible representations. Then, we create the DOM tree with Beauti BeautifulSon fuuso
4. Tax:: If the text contains markdown-style fenced code blocks, we use a regular expression to extract the contents within the first code block; otherwise, we treat the entire text as the output LaTeX code. We use a series of regular expressions to normalize the contents. This procedure removes the comments, excessive whitespaces, and commands such as [label. |cite, \eitep, /citet, \ref, \eqref, and \pageref.
5. Image2SVG Tex12SVG: If the text contains markdown-style fenced code blocks, we use regular expression to extract the contents within the first code block; otherwise, we treat the entire text as the output SVG code. We parse the SVG code and generate a PNG image with the ca irrsvv library.
6. Chart2Markdown If the text contains markdown-style fenced code blocks, we use regular expression to extract the contents within the first code block; otherwise, we treat the entire text as the output Markdown code. We use regular expressions to remove the code's comments, links, and excessive whitespaces.
7.
Chart2Caption, GUIJUserIntent, GUI2Summary, GUI-VQA: We directly evaluate the generated texts against the ground truth.
A.6.2 METRICS
Here, we provide brief explanations of BigDocs-Bench evaluation metrics.
Flowchart Triplet F1: Designed for Image2Flow tasks, this metric evaluates the accuracy of node relationships in flowchart codes, focusing on the correctness of edge triplets (s, e, d) extracted from Graph Viz Viz or JSON representations. Here, e denotes the edge label (set to None if unlabeled), while and d represent the source and destination nodes, formatted as "label#tshape". The Triplet F1 score is calculated by comparing the generated triplet list against the ground truth, disregarding the ordering of nodes and edges emphasize relational accuracy.
2. HTML DOM Tree Edit Distance: Applicable to Screenshot2HTML tasks, this metric measures the similarity between generated and ground-truth DOM trees using the Tree Edit Distance. Leveraging Beaut: i if sousp with 1xml parsing and the z.s library, the distance is normalized by the larger node count of the compared trees. Invalid Graph Viz JSON generations receive a Triplet F1 score of 0.
3. TeXBLEU (Jung et al., 2024)4: Utilized for the Table2LaTeX task, TeXBLEU employs LaTeX-trained tokenizer and a finetuned embedding model with positional encoding. Unlike traditional BLEU, TeXBLEU assesses similarity based on n-gram token precision, demonstrating higher correlation with human evaluations of LaTeX math expressions compared to BLEU, SacreBLEU, and Rouge (Jung et al., 2024).
4. RMSF1 (Liu et al., 2022);: Used for the Cardd2Mar task, RMS Fl treats tables as mappings from headers to values, measuring textual and numeric similarity through normalized Levenshtein distance and relative distance. This approach ensures robustness against row and column permutations and transpositions, making it well-suited for evaluating flexible Markdown table structures.
Availaabe at https: / com/ Foom/KyyDaan//// /Tt//LEX...... Availabbe at https:///gg /ttp:////////tb/////// com/ e research/ reroogggeeereeerch/treeeee ma master/deplot.
28

---

Published as a conference paper at ICLR 2025
5. DINOv2Score: Employed in the Image2SVG task, this metric calculates the cosine similar- ity between representations the ground-truth and generated images using DINOv2 (Oquab et al., 2023), which better captures image similarity than comparable models Invalid SVG generations are assigned a DINOv2Score of O.
6. Rouge-L, F1 (Lin, 2004): Applied to summarization and VQA tasks, Rouge-L. F1 measures the longest common subsequence between the generated and reference texts. We compute this score using the implementation provided by torchmetrics.
A.7 MAKING BIGDOCS LICENSE-PERMISSIER PERMISSIBLE
We dedicated significant effort to acquiring a license-pente permissible dataset suitable for training models for commercial purposes in document images. We aim to create a large-scale dataset that supports various tasks relevant to companies while adhering to accountability, responsibility, and transparency principles. To achieve this, we thoroughly investigated all public datasets concerning their licenses, evaluating both the sources of the images and their annotations. Our complete analysis, summarized in Table 5, enabled us to identify and retain only the sources that meet permissive licensing criteria.
Dataset licensing frameworks are crucial in determining how data can be used, shared, and modified, generally falling into two categories; permissive and restrictive licenses. Permissive licenses offer the most freedom, allowing for broad usage and modification of the data. For instance, the CCO license places the data in the public domain, enabling unrestricted use, modification, and distribution. The MIT License permits both commercial and non-commercial applications, provided the license terms are retained in any distribution. In contrast, the Apache 2.0 License extends these freedoms with an additional grant of patent rights.
In contrast, restrictive licenses imposes certain limitations on data usage. The CC BY license requires users provide attribution to the original creator. In contrast, the CC BY-SA license demands that any derivative works also carry the same licensing terms. More restrictive options, like the CC BY-NC and CC BY-ND licenses, prohibit commercial use and modifications, respectively. In cases where the licensing terms are unclear, it is prudent to exercise caution in using the data, as misinterpretation can lead to legal risks.
Additionally, the concept of Fair Use allows for limited use of copyrighted material without explicit permission, particularly for purposes such as research, criticism, or commentary. However, Fair Use does not equate to unrestricted permission, and its applicability vary, necessitating careful consideration when applied to dataset usage in research contexts.
A.8 HUMAN EVALUATION
To ensure fairness and minimize bias, we invited evaluators from diverse backgrounds, including PhD researchers, ML practitioners, multimodal AI experts, and individuals from both technical and non-technical domains. Key authors were excluded from participating to avoid conflicts interest. Some non-key authors, such as advisors and authors who performed smaller tasks on the data creation effort, participated. These authors did not know the performance of models on the proposed tasks. For the evaluation, we used our own web interface, make sure that all participants were anonymous and model outputs were randomized to prevent pattern recognition. We are open to releasing the detailed human evaluation results, as a supplementary document.
We developed a web application using Flask, Javascript, and HTML/CSS where users are presented with pairs of outputs from two models and are asked judge which model has better output for given input image. See Figure 12 for snapshot of the platform.
A.9 DETAILS UNIFIED METADATA FRAMEWORK
We propose a unified metadata framework for the BigDocs dataset to ensure transparency and traceability. This framework is organized into three primary keys: cicesse origin, and an optional features section. By adopting this structure, provide standardized and flexible system that enhances the dataset's 's usability, ensuring clarity for research and commercial applications.
hhttps: //me///mmmmm um. com / wwwkm//kkkhhooomm wtt.ccclliiiii-eeee novv2 in-iniime t-iaage--s aimi larity 6frity-6faaaa7ddBa
29

---

Published as a conference paper at ICLR 2025
Table 5: Datasheet of candidate datasets. For transparency purposes, provide detailed description of over 100 datasets considered in curating our BigDocs dataset. This table presents a systematic analysis of public datasets, including information on medium types, source documents, text structures, languages, years, annotation types and methods, and licensing for both data and annotations. We also share sample counts across different modalities and splits. Some fields may be blank due to wavailable information. This comprehensive overview enables assessment of each dataset's characteristics and potential contributions to BigDoes.
Bado Suig
Cadoad
13
Along with the dataset, we include two separate files named and tracking t rans fo Format ioss. json, which are dictionaries that store information stan- dards across large subsets of samples, reducing redundancy and promoting consistency in metadata.
conssruccions json
License Metadata The 1 iicense key is mandatory and provides detailed information regarding the licensing terms of both the images and annotations. It is structured as a dictionary with three sub-keys:
mmage_lccent (mandatory): Specifies the license under which the image is provided.
annotat ion_ con_licensse cicesse (mandatory): Indicates the license for the annotations or labels associated with the image (e.g., text, bounding boxes, tables).
sicense_nooi note (optional): Includes additional licensing information clarifications, such as specific usage conditions or exceptions. This allows for clear communication of any nuances in licensing terms.
30

---

Published as a conference paper at ICLR 2025
AudgeCalts
on
Which Outpuris Berter?
Result
Result
Al:
Miche smade
Prompt
"draage (CC imhil image?
Utbar
"Jragg, ChCass image"
Mooki Oelorut (A1
thight
(2) Moree Outpet 180
your chatie
Oveis "Wlhowe anaryyoouu"
Figure 12: Human Evaluation Platform. is presented with model outputs and asked select the one that more accurate in relation the original model input.
Proposed metadata of data sample from TabFact dataset (Chen et al. 2020)
natadata: features: (image height: 1584, image width: 1224j, license: Jannctation license: OC BY 4,0, insge_license: CC BY-SA 4.0). origia: [table_peth: bata/all_cen 2-2---4016755 esv. data_id: 2-15401676 115t0166--..tmm.... cew, rataset_uzik hetpa /ttpt/////ataa tabta bactt,tactiia trakkingaacnsg arrggaaaaasssanttteeeeeee formationn tabfast_i]
wwt.a//wwwww.ccccee.e.c.. www.wwwnuchee//eee.... www.---eeetteeeeeeeeeeeeee o-Fact-Chsssknng
tracking instructions, jeom: tabfact: "Clone repo from setasse_urll loaae CSV via teble_path, ronler inage."
tracking, transformations jaon:
ID
Description
Labfact 1
Use table tite as intage caption.
tabfact
Concalenate annotation ficts as summiry.
Eabfact_
using esv, convert markdown.
Figure 13: Metadata framework from BigDocs Toolkit for sample in the BigDocs 7.5M dataset. includes details about image properties, licenses, data sources, and transformation steps, such as using the table title as an image caption.
Origin Metadata The or ori iign key is mandatory and serves as a traceability mechanism, allowing users track each data sample back to its source. This key is also structured as dictionary and includes the following sub-keys:
data id (mandatory): A unique identifier for each data sample (e.g., file name, index, or ID in the source dataset).
id (mandatory): reference to an entry in the json file, specifying how to use the metadata fields to trace the data sample back to its source.
braccking_instrrooe
(optional): A reference to an entry in the onss. json file, providing a high-level description of any transformations or modifications applied the original sample.
dataset_uris (optional): The URL of the original dataset from which the sample was obtained, if applicable.
The local path the image within the raw dataset files, available. This key is helpful when the dataset is stored in raw form.
image_pate (optional):
image_ur (optional): The URL from which the image was initially downloaded.
Mi sc. (optional): Other information related the origin of the data entry.
31

---

Published as a conference paper at ICLR 2025
Features Metadata The features key provides detailed characteristics of the data sample. This can be useful for downstream tasks that require specific information about the sample, like the image size and type of annotations.
heige_heighte The height of the image, in pixels.
wtthe_width: The width of the image, in pixels.
Common Metadata
brnckinngiistttuttiii json and tracking wainnngtrassneee cransformmtiins json files are designed to store information common many samples, minimizing redundancy and ensuring consistency, Each file is structured dictionary:
Files.
The
trackingginae wew.iinsssuuiinsss json: Contains instructions on how to use the metadata fields (such data id, dataset aastsst__ur,, etc.) trace the data sample back to its source. Each entry is uniquely identified and can be referenced in the individual sample metadata by tracking_instructt.om
tracking sracking_ttransfomee wtw.ttrannooraaaaoammm processing steps, or additions to the original samples. This includes synthetic gen- eration processes, data augmentation steps, other modifications. Each transforma- tion is uniquely identified and can be referenced in the individual sample metadata by tracking_tran formation cirmmaiionndd
json: Contains descriptions of any transformations,
A.10 QUALITATIVE RESULTS
We provide detailed qualitative analysis to highlight the performance of different models tasks within BigDocs- BigDoes-Bench. Bench. Table 6 showcases the outputs of our best-performing model, Phi-3.5- Phi-3.5-Vision trained BigDocs, alongside publicly available instruction-tuned model and GPT4o. The analysis covers tasks such as GUI-VQA, Chart2Caption, Image2Latex, and Screensho2HHTML Notably, the BigDocs-trainee model demonstrates more concise and accurate free-form outputs for VQA tasks, generates perfect LaTeX tables from table images, and produces precise HTML code from web screenshots.
Vision
Additionally. we present a series of qualitative examples across BigDocs- Bench tasks, detailed in Tables 9-28. These examples provide direct comparisons of outputs from all models discussed in the paper. The examples are accompanied by a thorough error analysis that highlights each model's strengths, limitations, and both success and failure cases.
A.11 GLOSSARY & TASK DEFINITIONS
We facilitate list of concepts introduced and utilized throughout our research. Below is a list of these concepts and tasks, along with their definitions, including descriptions, inputs, and outputs.
Text with Structures
Description: Analyzing documents combining text with elements like headers, lists, and paragraphs to understand the overall structure and organization.
Input: Documents containing structured text elements such as headers, lists, and paragraphs. A structured representation of the document's content.
Output:
2. Structures Figures
Description: Identifying and interpreting figures, diagrams, and other visual elements in documents to extract the information they represent.
Input: Documents with embedded figures, diagrams, and visual elements.
Output: A detailed understanding or extraction of the visual content.
Infographics Covers
Description: Examining infographics and cover pages that use mix of text and graphics to communicate information visually.
Input: Infographics and cover pages containing text and visual elements.
32

---

Published as a conference paper at ICLR 2025
Table 6: Task Structure and Samples with Model Ouiputs for Phi Models and GPT-40
Phi- Phi-3.. Instrnct- Untrrnct-Bighhos Hlgghho
Ph--L. Vidton Instrned
Task
Query
GUI-VQA
[The image not provide mation on what opens when the user clicks the specitted coordinates]
[View the news]
[By coondinates (620, 50, 70), the weer will navigate the "news" section the web- site.
[Whal upens when the user clicks 1550) 8017)
Chari2Caption
image displays various segments. repessenting difterest range values cormqunnding percentages arp- are colee-oodeed with both the value range und the percestigg they expresent. The chart is snuneed from Statists 2024]
[Dittibation
Levels: Percentage
caption image,
range employees United Kingdom Brilish pounds, parcent comployces falling into this bracket. Employees carning tween 2010,000 and 249.999 Brassh
Households Bracket 3024]
plesse]
pounds. made up the second moot common salary range, 15,8 per cens.]
1300
Image2Latex
(Clente LaTeX code prodice this table.
ScLL
[Coald you create Hly penest Madcup Lan- code from the screenshor!
Output:
A comprehensive analysis of the combined visual and textual information.
4. Structures Tables
Finding and understanding tables in documents, focusing accurately captur- ing the data and relationships they present.
Description:
Input: Documents containing tabular data.
Output: A structured dataset or representation of the tabular information.
Structures Forms
Description: Recognizing and processing forms within documents, extracting data from fields and labels typically found in structured forms.
Input: Documents with structured forms, including fields and labels.
Output: An organized collection of the extracted form data.
Natural Image
Analyzing photographs or natural images in documents, interpreting their content in the context of the surrounding text.
Description:
Input: Documents containing photographs or natural images.
Output: An interpreted categorized representation of the image content.
33

---

Published as a conference paper at ICLR 2025
A.12 ABLATION EXPERIMENTS
A. ABLATIONS ON DATA CURATION
We ablated the main data curation procedures on BigDocs, namely the use of 1) VQA format during CPT, i.c. using (question, image, answer) tuples with assistant and user conversation formats, 2) the use of heavy OCR tasks during CPT (converting most of our document images into OCR tasks) and our 3) text-image alignment tasks, mainly captioning of many datasets. Experiment Setup. Using
Table Ablation Experiment on effect of VQA formatting, presence of OCR data, and presence of captioning data in BigDocs-7.5M. Results show the imporiance of including these curation steps and tasks. See discussion in Section A.12.
Sldde VOA-M
Voe
VOA
Desp DospForm Form rasr
TabFact
Chari
Te
MMMU
DudeMMai
TablVVO
Avg Score
un
KLC
WTQ
mar
VEST
Model
Original
87.05
70.05
70.97
37.45
51.21
81.24
81.56
68.72
45.00
36.15
32.47
67.77
60.80
82.65
28.8
36,93
42.33
No VQA
(-44)
(-15.13)
(-30.33)
(-865)
(-14.28)
(-15.94)
(-3.16)
(-3.48)
(:2.67)
(-1.44)
2(-4.1)
(-6.64)
(-9119
54,19
34.9
$2.68
72.07
42.56
32.07
5%.1
30% OCR
(-1.26)
(-10.02)
(-16.78)
(-2.55)
(1.47)
(-7.48)
(0)
(3.35)
(:2.44)
(0.57)
(-0.4)
(3.06)
(.2.71)
85.36
57.98
53.36
35.09
50.91
71,92
31,97
69:37
57.38
30% caption
(-1.69)
(-12.07)
(-17.61)
(-2.36)
(-0.3)
(-8.74)
(0.84)
(3.2)
(-1111)
(0.61)
(-0.5)
(1.6)
(-343)
the Phi3.5-Vision Instruct model, we created three modified versions of BigDocs-7.5MM one without the VQA format, one with OCR tasks removed, and one with reduced captioning tasks (text-image alignment). Each dataset variant was used to train the model, followed by fine-tuning the model on our instruction-tuning datasets. We evaluated these models on the benchmarks listed in Table 2 (General Document Benchmarks). The results are presented below. In parentheses, we show the delta with respect to the original result.
Main Findings.
1. VQA Format: VQA formatting proved crucial, as its removal led to a 9.19% average performance decrease. This especially apparent in InfoVQA (-15.13), DeepForm (- 30.33), WTQ (-14.28), and TabFact (-15.94). A common pattern observed here a lack of adherence to the requested output format as demonstrated in the example in Table X. We hypothesize that the VQA format used in the continual pre-training stage (CPT) enables better instruction tuning and supports multitask generalization.
2. OCR Impact: Removing OCR tasks led to average performance drop of 2.71%, with varying impacts across benchmarks. Notably, we observed significant declines on Info VQA (-10.02), DeepForm (-16.78), and TabFact (-7.48), while results on DUDE, SlideVQA, and ChartQA remained largely unaffected. The reduction in OCR data increased hallucination in tasks requiring precise information extraction. For instance, in DeepForm, the true negative rate dropped from 62% to 0%, as models began hallucinating unrelated values from other parts of the document. Examples illustrating this behavior are provided in Tables X and Y. Interestingly, in some cases such as Tab (+3.06) and TextVQA (+3.35), performance improved, likely due to the nature of the question-anse answering tasks, which often favor free-form text generation over exact OCR matches. Table 21 shows example of this phenomenon.
3. Text-Image Alignment: Reducing text-image alignment tasks, particularly captioning, resulted in 3.4% performance drop, with notable declines on VQA (-12.07), DeepForm (-17.61), and TabFact (-8.74). This drop primarily stemmed from errors in parsing values for arithmetic operations, especially when data formats varied, such as converting text or unique visual elements into percentages illustrated by examples in Table X. Table Y and Table Z. Tasks without significant drops, like WTQ and ChartQA, involve simpler text-visual correspondences that don't 't require complex conversions. See Table 22 for a qualitative example
34

---

Published as a conference paper at ICLR 2025
A.12.2 ABLATION ON CONTEXT LENGTH
The results presented thus far have been limited to a context length of 8192 tokens. Given that the ScreenshotHHTMLL task averages around 32k tokens per example (as shown in Table 1), it encounters significant challenges under this 8k-token constraint. This task ranks among those with the lowest scores (see Table 3), which contributes a decline in the overall aggregated average score.
Despite this limitation, we found that the generated HTML code remains compilable, and the resulting DOM can still be evaluated using the DOM Tree Edit Distance metric, even though the HTML output is truncated due to the context limit. However, this inability to handle complete HTML outputs highlights the necessity for models capable of processing extended context lengths.
To better understand this issue, we analyzed the model outputs (HTML code) in terms of a histogram of token lengths, comparing them with the ground truth lengths in the test set (see Figure is important to note that the test set is designed so that the images can be represented in fewer than 8192 tokens (as shown in the left figure). This figure highlights that token length constraints can adversely affect the task, as most samples approach the maximum token length of 6k (after accounting for image tokens), as seen in the right figure. Although the test set is designed to be generated within a reasonable token length, the model's training on larger contexts leads it to attempt to generate longer HTML outputs.
We fixed the context length at 8192 tokens for several key reasons. First, not all models evaluated in our benchmark natively support larger context lengths, and extending this limit could disadvantage some models. Second, increasing context lengths significantly boosts GPU memory usage, compli- cating both training and inference at scale. Finally, to ensure fair comparisons across all models, we standardized the context length 8k tokens.
We conducted an ablation study on context length by training the Phi3.5-Vision Vision model with a context length of 32k tokens. Subsequently, we re-evaluated the Screensho12HTMI. task using context lengths ranging from 512 to 16k tokens. However, generating 32k tokens resulted in out-of-memory errors. The results are presented in Table 29, where we applied greedy decoding (second column) and beam search with a width of 2 and length penalty of -1.0 (third column).
The results are revealing. We do not observe any performance boost when training and evaluating with longer context lengths. The greedy decoding algorithm performs worse with longer contexts, not exceeding a score of 10.99, In contrast, beam search with a length penalty approach yields better results, reaching 15.80, but still does not benefit from larger context lengths.
Additionally, we conducted qualitative error analysis (see Figures 19 -22) and found that the model often struggles with stopping generation, frequently hallucinating repetitive patterns such as nested <div> elements or excessively long URLs and references. However, well-formatted HTML code is generated in approximately 40% of cases (Figure 18, top figure). Applying length penalty helps mitigate this hallucination, leading to improved performance. This indicates further potential for enhancing results with alternative generation techniques.
This indicates that the performance issue may not stem from context length but rather from the inherent difficulty of the Screenshor2HTTML task. This aligns with the observation that the test set's average context length is approximately 3k tokens (Figure18, 8, left) and does not require more than 8k tokens. We suspect that the primary limitation lies in the insufficient amount of training data (currently around 10k samples) which may hinder the model's ability to learn HTML generation effectively.
A.13 DETAILS IN DATA VERIFICATION ON BIGDOCS-BENCE
This section provides a detailed description of the verification methodology, evaluation criteria, and measures taken to ensure the quality and reliability of BigDocs-Bench. We outline the qualifications of the human annotators, the multi-step verification process combining automated filtering and human review, and the strategies used to address the challenges posed by the dataset's 's large size and synthetic nature. The section also discusses our sampling- based quality assurance for the training split, inter-rater reliability measures, and the conservative approach adopted to prioritize recall and dataset safety. These steps collectively demonstrate our commitment to maintaining high standards in creating robust and ethical benchmark.
35

---

Published as a conference paper at ICLR 2025
A.13.1 VERIFICATION METHODOLOGY
We generally follow a 4-step strategy perform human verification as follows,
1. Automatic Filtering Tools.
We implemented multiple automatic filtering tools as the first step in our quality control process. These tools were designed detect issues such bad annotations, NSFW content Alama-Guard-- from https:/huggingface co//meta-m wwmmmttalllama/L... llamm/Llama- Saamma-Gaard---11 IB- B-Vision), and Personally Identifiable Information (PIT). This automated process reduced the volume of problematic samples before any human verification was performed.
Human Verification for Test Split. To ensure high-quality test split, we only do human verification focused on the test set our automatic filtering is already quite robust. Annotators identified and removed problematic samples following the guideline, for example, unnecessary comments in LaTeX and HTML code (e.g., in Table2LaTeX and Screensho(2HTML tasks), which could cause rendering issues. Similarly, flowcharts with excessively isolated nodes tasks like Image2Flow were flagged and removed. For Image2SVG and three GUI-related datasets, spotted and removed those images with PII or NSFW content. All these typical errors were collected and documented during this process inform our automatic filtering tools.
3. Refining Automatic Filtering Tools for Training Split. The typical errors identified during human verification of the test set were used to iteratively improve our automatic filtering tools. This enhancement enabled the tools to better detect and handle similar issues in the training split, which was impractical to verify comprehensively through human labor. 4. Sampling- Verification for Training Split. To ensure the training split met quality standards, we randomly sampled 100 samples from all tasks, proportional to the size of their respective training splits. These samples were verified by annotators based on the criteria mentioned below, and the overall pass rate was 99% (i.e. 99 good samples), reflecting high level of quality for the training data after applying the refined filtering tools. Specifically, every task has pass rate of 100% except GUI-VQA, which has a pass rate 92% (i.e. bad sample out of 13, which is not relevant to the task since the question is not quite related to the given image).
Human
A. VERIFICATION CRITERIA FOR VERIFIERS
The verification process for BigDocs- Bench prioritized several critical aspects ensure the dataset's quality, reliability, and suitability for real-world applications.
1. First, data integrity was primary focus, ensuring accurate alignment between inputs and outputs. For example, in tasks like Image2SVG, visual elements images were verified to match their corresponding SVG annotations, ensuring precise vector representation. Simi- larly, for Screenshot2HTML. rendered HTML outputs were checked against the screenshots confirm structural and semantic fidelity.
2. Second, the relevance of synthetic samples was thoroughly assessed to confirm that they were realistic and reflective of the intended task objectives. For instance, in the Image2Flow task, flowehart samples were evaluated to ensure that the generated ISON or Graph Viz code meaningfully represented the logical flow of the diagrams without excessive isolated nodes disconnected components, which could compromise their utility.
3. Finally, ensure content safety, NSFW content and PII, such explicit material, or personal data, were removed.
A.13.3 DETAILS ON HUMAN VERIFIERS
We engaged team of 6 human verifiers, consisting of graduate-level researchers and professionals with expertise multimodal datasets and document analysis. All human verifiers were thoroughly trained on the verification criteria outlined below before commencing the review process. Each data sample was reviewed by at least two verifiers to ensure thoroughness.
A. 13.4 INTER-RATER RELIABILITY
In the test split of our benchmark, we prioritized recall to ensure no potentially problematic samples missed. For tasks like filtering NSFW content, detecting annotation errors, identifying
36

---

Published as a conference paper at ICLR 2025
Table 8: General Document Benchmarks. Models trained on competitively across multimodal document benchmarks. compare them base checkpoints, instruction-uun tuned models, and those trained on (DocStruct4M Downttrrrr. We also report scores on state-of-the-at models > 7B for reference. BigDocs models show consistent performance across tasks. Refers the use of Chain-of- Shain-of-Thought (nough((Col)
perform
Slidk VOA-M nis
Dhe
Jn
DeepForm
KLC
WTQ
TabFaet
ChariiA
MMMMU
DudeMMal
Tahle ytor
Ang. Score
Model
DocOwll1 (instnact)
80.73
49.94
68
37.99
38.87
79.67
68.91
33.67
31.62
$2.60
$3.84
DazOwit 5-88 (hase)
2,07
1.84
0.00
0.00
0.001
0.00
0.00
0.00
24,44
19.07
3.30
13.63
5.36
DecOwit 1-3.1.5.48 5:4B (base) D
75.99
46
62.77
35.21
32.86
71.56
68.36
65.08
33.67
29.00
27.00
46.27
49.56
DocOwli 5-88 (base) BigDoes (Ous)
78.70
47.62
64.39
36.93
35.69
72.65
65.
65.80
67.30
32.33
32.55
29.60
49.03
$1.08
Qwen2-VL--22 2B (instraze)
89.16
64.
32.38
25.18
58.20
57.21
73.40
79.90
42.00
45.23
43.07
53.03
Owen2-VLL-2B (base)
7.26
0.78
0.00
0.00
0.00
0.00
0.00
1.14
34.89
28.43
14.55
0.00
7.25
Qwen2-VL- (base)+ DocStrucsMM
59.53
32.00
53.98
36.38
28.48
64.24
54.44
55.89
34,89
28.78
22.68
46.53
43.15
Qwen2-VL-2B (base)+ BigDocs (Ouns)
57:23
31.88
19.31
31.39
31.61
64.75
68.60
61.01
35.67
27.19
17:46
47.53
43.89
Phi3.5. Visioo- (instract)
36.20
10.47
7.49
17.18
73.12
45.66
Phi3.5 Visiio-Visioo-- 4B + DocSStuetIM
86.76
68.90
70.12
37.83
$1.30
$2.12
79.76
68.60
44.11
35.52
31.90
69.17
60.51
Phi3,5-Visionn 1B + BigDoss (Ours)
87.05
70.05
70.97
37.45
51.21
$1.24
81..6
68.72
45.00
36.15
32.47
67.77
60,80
TLaVAcNeXT-7B5 (imiiccc))
63.51
30500
1.30
3.35
200.16
$2.83
52.12
10
38.83
17:94
7.46
32.87
32.36
11.VA-NeXT-7R- DooSirac14M
60.95
26.14
19.78
28.34
LLaVA-NeXT-7BB BigDoes (Ours)
57.13
24.17
46.38
31.09
27,06
72.58
51.72
19.06
17.78
22.88
16.07
33.13
37.70
Llama-32.908
74.15*
48.71
4.18
1.81
24.20
63.01
11.36*
71.69
57.78
41.24
26.09
41.57
38.82
GPT-4o 20240806
92.80
66.37
38.39
29.92
26.63
81.10
85.70
70.46
69.10
54.55
67.55
72.87
64.62
Claude 3.5 Sonnet
88.48
59.05
31.41
24.82
47.13
53.48
51.84
71.42
64.78
35.11
81.27
50.73
GeminiPro-1k Qwen2-VL-72B
73.94
32.
21.07
$0.29
$7.05
96.50
84.50
30:45
24.78
55.63
0.00
88.30
85.50
64.50
35.87
2.15
74.23
58.40
corrupted data, any sample flagged as problematic by single verifier was removed. This conservative recall-focused approach maximized dataset quality and safety while addressing misalignment between
raters.
14 RESULTS ON LARGER MODELS
We have evaluated larger models on general document benchmarks, and the results are presented in Table 8. Models with more than 8 billion parameters perform exceptionally well the general document benchmarks leaderboard. The top performer is GPT-4o, achieving an average score of 64.62, followed by Qwen2-VL-72B with 58.40 and GeminiPro-1.5 with 57.05.
A.15 CORRELATION ANALYSIS OF BIGDOCS -BENH WITH OTHER VLM BENCHMARKS
To assess the novelty of Big Bench in the VLM landscape, we conduct a correlation analysis that highlights how BigDocs- Bench distinguishes itself from existing benchmarks. Specifically, we leverage model results across VLM benchmarks to demonstrate that BigDocs-Bench -Bnch offers unique dimensions of model evaluation and analysis.
First, we analyze the correlation between Document Benchmarks by comparing the results from Table 2 and Table 1. Following this, we expand the analysis include results from several widely used VLM benchmarks, providing comprehensive view of the relationship between BigDocs-Bent Bench and prior benchmarks in the field.
Analysis Setup. We selected all benchmarks listed in Tables 2 and 1, encompassing both General Document Benchmarks and our proposed SegDocs-Benchhe respectively. We constructed matrix encompassing all models presented in the paper, namely DocOwl-1.5-8hh
Aive22-V-22B LLaVA- NeXT-7B, Phi3.5-v-4BB v-B, both the off-the-shelf public versions and the ones trained on BigDocs. We also extended Table 2 with scores for GPT4o, Claude 3.5 Sonet, Qwen2-VL-72B, GeminiPro-1.55 and -..ma-3.2-4
to obtain a full matrix. Note that we have now extended our baselines with Llama-3.2-905 Finally, aggregated results for all metries in the two groups of benchmarks. We have edited our manuscript to include Table 8, which is an extension of Table 2 with scores from all models.
In our broader study with general VLM benchmarks, including but not limited to documents, we expanded the analysis to encompass 28 additional VLM benchmarks (see Figure 16 for the full list). Missing scores were imputed using the mean to complete the matrix for models and benchmarks.
Methods.
37

---

Published as a conference paper at ICLR 2025
vo
3000
Clung
Twrrus
K10
Deghoen
Imgothe
Card
Hights Beat (feg)
Imp
Figure 14: Correlation matrix and dendrogram for BigDoes-bench and General Document Benchmarks. The matrix (left) shows benchmark relationships, with BigDocs- Bench tasks (circles) clustering separately from other General Document Benchmarks (triangles). The dendrogram (right) highlights distinct groupings, including BigDocs-Bence tasks for code and GUI generation clustering together and VQA tasks forming separate cluster. Tasks like DeepForm and KLC align closely, reflecting their shared complexity.
Cosine similarity matrix and dendrograms: We normalize metric scores by centering (subtracting the mean) and scaling (dividing by the Euclidean norm). We then apply hierar- chical clustering using cosine distance to group benchmarks and visualize their relationships. Cosine similarity matrices and dendrograms illustrate these clusters.
2. Principal Components Analysis (PCA): We represent each benchmark as feature ar- ray composed of scores from all models and project these arrays onto the principal dimensions.
A.15.1 CORRELATION WITH DOCUMENT BENCHMARKS
Our results shown in Figures 14 and 15 show that SreDoos--Bencl tasks and benchmarks are different from every other document dataset.
Figure 14 presents the correlation matrix and dendrograms. Circles represent BigDocs-Bent tasks (our benchmarks), while triangles denote other document benchmarks included in the paper. These results demonstrate that BigDocs-Benna Bench tasks are notably distinct from other benchmarks, as indicated by their low correlation with them. A clear grouping emerges for tasks related to VQA, as well as LeDDocs-Bench tasks involving code generation in LaTeX, JSON, Graph HTML, and SVG. Notably, HTML and SVG tasks form cluster, likely due to their characteristically long output sequences. Additionally, DeepForm and KLC stand out separate cluster, distinct from all others.
Figure 15 illustrates the PCA results. The clusters distinctly separate BigDocs- tasks from other benchmarks, reaffirming their unique characteristics. Notably, the average aggregated score from BigDocs- BigDocs-Bencc (left side) is the furthest from the aggregated score of other benchmarks (right side), highlighting the distinctiveness of our benchmarks. However, certain tasks, such as Image2SVG and Sereenshot2HTML show stronger correlations with KLC, DeepForm, and TabFact. This is likely due to the shared level of difficulty across these benchmarks.
38

---

Published as a conference paper at ICLR 2025
AKIC
0.50
ANTO
PCA Component
0.25
SEACTOON
Dae Beadomanto LAng
0.00
-0.25
Letim (Cepring TekiLan Putther
-0.50
C CEUSsa
Document Benchmarks
-0.75
BigDocs Bencch
PCA Component
Figure 15: PCA of BigDocs-Bench and General Document Benchmarks. BigDocs- Bench (circles) form distinct clusters from other document benchmarks (triangles), emphasizing their unique characteristics. Tasks related code generation, such Image2 Flow and Late, cluster together, while VQA tasks, including those from other benchmarks, are farther apart.
A.15.2 CORRELATION WITH GENERAL VLM BENCHMARKS
Figures 16 and 17 show that BigDocs-Bench Bench tasks and benchmarks are unique in the broad VLM space.
Figure 16 presents the correlation matrix and dendrograms. Circles represent LeDDocs-Bendde tasks (our benchmarks), upward triangles denote other document benchmarks included in the paper, and downward triangles denote general VLM benchmarks. BigDocs-Bee -enchh tasks demonstrate distinctive evaluation characteristics compared to existing benchmarks in the VLM space. This is evidenced by the clear clustering pattern, where BigDocs-BBnchh tasks form their correlated group while showing minimal correlation with other benchmark types. Results also reveal three main clusters of tasks: general vision-lang language tasks (like COCO VAL and VCR), document understanding tasks, and GUI/talle-related tasks (such as our tasks GUI2Sum and Tablc2Latex). This clustering pattern suggests that BigDocs-Bees -ench addressing unique aspects of document understanding that isn captured by existing benchmarks.
Figure illustrates the PCA results, revealing that most BigDoes- Bench tasks are clustered in the bottom-right corner, distinctly separated from other benchmarks. Notably, tasks within BigDocs- Bench form clear clusters based on their type, such as VQA, Flows, Charts, or GUTs. General VLM benchmarks create dense cluster, while document benchmarks more sparse. Additionally, the aggregated BigDocs-Bent Bench average score is positioned far from other benchmarks, further highlighting its uniqueness.
39

---

Published as a conference paper at ICLR 2025
Seoooo WE-N 16
100 1000
De Veciio
of Howe
volteut TEST
Doe Resdinati
LLaVAbont
005300
Figure 16: Correlation matrix and dendrogram for BigDocs-Bench and General VLM Benchmarks. The matrix (left) illustrates benchmark relationships, with BigDocs SigDocs-Bench tasks (circles) clustering separately from other benchmarks. The dendrogram (right) reveals three main clusters: general VLM tasks (e.g.. COCO, VCR), document understanding tasks, and GUlttble--rebeee relattd tasks (such as GU12Sun and Tatle2Latex).
Ducument Benchmarks
FOOCOLMM VAL
BigDocs-Bena Bennch
9
OPe
1.0
PCA Component
XMCleck TEST
0.5
SaceooA TesT
0.0
Au2o
GEEARA
3 TEST.N JNCVI
TATYOA
V Pidaace
O SLttIe
-0.5
9 MEAoe
Ma
TakeTan
Chan
Eaupdiio (lot)
McDred T1ST CR
-1.0
Hullas Harik
Dupplee
Component
Figure 17: PCA of BigDocs-Bench Bench and General VLM Benchmarks. BigDocs Bench (circles) form distinct clusters, clearly separated from other benchmarks, highlighting unique characteristics. dense cluster observed for general VLM benchmarks (downward triangles), while document benchmarks (upward triangles) are more dispersed.
40

---

Published as a conference paper at ICLR 2025
Table 9: Test Example for the task of Char22aption. All models other than Phi3.5-V-45 make mistakes in the caption. LLaVA -eV---NeT--7B does not offer relevant information about the chart, but about its format and + BigDoes hallucinates. BigDocs models make factual mistakex, while Claude and GPT40 have trouble with the accent marks.
Compunent
Content
Suriger feleer
10.99
11:00
24,00 160
2 Co lapos stotbils
Enput Imagee
User Query
Can you give me a caption for the chart in the imaaee
Grund Truth
nivel contavins (CDWIDs19) canvell cartii number deaths within the Franch popolationn With vietins, the Regiun, drndeeernne, recorded the highest mumber of deaths in France as of June 30, 2025 the other hand, the regions of metropoltzon Frazce lest affected were those of Britary and Coescaa Fhr tutber information about the cororavinus (CVVID 19, pandemic please vail our dedicated
Facts Figurs page.
V-43
horizontal gruph dispinyteg the mumber doaths sarlow regions France. xosis represeios sumber regrons Heede-France yrspl 2024.
doants ronglas from 17,750, Grand Ear
highest of descendingg
25,101, followed
deaths
Corsia 538,
indicated
Phiks-V-IR Higthess
2022, deportsent He-de-Frince recorded the highest somber deattr France, with 29,001 deatlir, The department Haattee- France reconded the second highest number deallu, wish 13,907 deaths.
LLAVANNTTB
The chon the image is bar graph sidvdd "Callape mtalatic" and & slous the namber of deatle for vorisur fovations in Fancce The locations
lited on the loft side of clurr, and the susber of desths is eppesented by the neight of each hor The batz coler- and the thart welade a legendi lo expiein the color coding The chart olo har bubtto labeled "Cullipse itatitic" the betton, suggesting that the suer can inserat with the chart to collapue or cqaadd the dass,
LLaVA-NNN1-7- Bightoss
Since mar of French Kesolation 1791, oser 2MM people bave been execated in Prance. The must commun canse of death was by shopp
guiliotins, procedure
aeuny to curvied oat October 2023, followise the drath of volired police offict
sever from the body. The must recess coccution was
Qwen2-V1.28
Nawber of
by region
Qwen2-VL-2B HigDocs
Franch depanivent wish the highest number of destla dus
(COVID- 19) Re-de-Fraccy, witk 29,107 deulks the iesting cupacily conduct PCR people
The Franck auhoviiiz
triing
gapposu onder and discmer exactrrat droth
DieOul-15-8B
The depariment with the highest mabber of deants France 2019 war Riedi-
France. Riedi-France
DieOol-15-88 BigDaes
In 2022, them ave 29, 101 duale in France. Of thee dhaaos 7,455 ver the Boargager
making the depantment with the highes
mumber of dooths to Fronce that year
Chande
A samuble coption for this chart rould by: Nanaber of COVID-19 Duaths by Region in France This boriosand bor okur shoees the samber of deatie, presumails from COVID-19, ocmss difirral regioss France, The regions are findd os the lef, with divviedde-Fune Frunce having the lighest number of deathe
1U), while Cardion has the lnnert 538 The dato appears to be presenting consinttee deam rells, Shely over the course the pandenic for coch of France's adsinimmatiss recinns.
GPT4.
Hamber of Deatl by Regies in France: Urriledi-Fanct Leads with 29,101 Deaths, Felloaed by Awwenmn Neweyn--Rise--aee Rase-Aipez and Grand Est
41

---

Published as a conference paper at ICLR 2025
Table 10: ChartCaption: Comparison of model responses showing that models using the BigDocs dataset produce more concise and relevant captions than baseline models while still making some factual errors.
Component
Content
Eaput Image
User Query
need captios for the chart the image, please.
Ground Truth
This graph slows made civlian frece purticipation ratc in the Unital State
In 2023, 68.1 percont of the
sidlian populaius putisciattall in the job manks.
V-43
zrap repereenss labor force ipaticpptioon
gus year from 1990 2023. vertical labited
Labho particuation rutt' seasared percentoges ranging from 2022, 2023. The line dovmvnn the labor 2020,
The
represouts years,
single-year poried. 2000 omsnnds roching lourst ther is sight increase followed by plateos ground 68% the must recest years dispayel.
apprndinaatt 28'2 19h0, experincor
farnations, then begine
60% 2030 After 201h
V-4B-- + Higlbaes
In 2013, 68.1 percent of the femole inbor force participated io the job marker. This it dlght incresve from the presious year. when 67.7 pereen of the Jeauir labor joice partt qpated in the job warket.
LLAVA-NXN-77
The shart in the imsgy slurs a lane grask with dossmaard nead indicating a decrsse is vertain metnt over fine. The suetric is labeled as "Labor Force Participation Ran,"" which is messore the namber of people who are clsher vorlivg or actieey antwllssrsing stoking cxpliryment as
perventoge of tonal papalation, The graph is ser agoiist is white pooigrousd with blar for syuranntig the donr points, and there any additional information icous the tpp right cornes suegestive
ioer
thes for more doulls options. The
"Additional Information" and includes "Show source" iist, indicating that the dow pourced Murista
LI RigTtes
2612, labor force participation descabe pervatage chillae one-iustiuniinal nun-qultury prpslation that ampplyye
United
percent
funee participation by
Labar
Qwm22-VL.28
from 1902 3123, The lobor fone participaiio rate the of the population thar rither explooed memolly teaking eapinymost The doto dbore siguifcant decline inbur fone particqution lute ovr the years, with nombii dro in the late 2009 and carly 2010..
The rhant depiers the lobor fonc porticipatios
the United foae
V1.22-VI.2R-44 + Highoes
In 2023, about 6855 percont of the tohor force particigutee in juhevlated activttiss the United Stotes, This decrneve from the presioor jear, wien about the prrcent of the lahar forv panicppttdd is job-mlated activites The lohor forez participaiins nat pasked in 2027, at 70.4 perceut
DieDuu-1.--0
The tlar for thart oppesoots loior foive purtcipation car the labar forc Unued States
DieOul-1.5-81 + HigDaes
In 2023, the labar force particppatinn rate the United Stater way 66.3 percent, Title is decrvase from the previour year, whbrs the tabar
foree partcipation rate mood at perceat.
captios chart
Later Porticipatos is United 1990-2022 WalTi caption accuuanety
partcipatiion 1998 2013, gruph showing: goncol declise luber partictpation
prsted of apprcitaally
76%
1950 ending
2023.
GPT4
Tronds Labor Fore Participation Rote (1900-2222) Decline with Mecant Snaiiltotion.
42

---

Published as a conference paper at ICLR 2025
Table 11: Evaluation Example for GUI-VQA: Models utilizing the BigDocs dataset consistently produce the most accurate and concise outputs, perfectly matching the ground truth without any extraneous or unnecessary information.
Compunant
Cuntenl
Input Image
Queey
[Whtt information appears whcc clicking (330, 300, 590)7]
Laundrette.
PhikS.V-4B
imuge doss prorisle information appears clicking (330, 590).
P-43. BigDbos
Access Laundrette.
LLANA-NeXT-7B
with services offered udnd anier differen rategorkes and lobve ricc the mardorr (149, 590, 1w: So0), anoom narigate differvat settion of the website documesii
pevidad appears avsesster sellyage
LLXVANNcXT-7B+ BigDoes
Access the Laundry
Qwen2-V1.28
When clicking on the "Moter Repair" batton, Ahe information that appears is related to the malistronnce and rrpair moter spoveus.
Qwen2-VL.2B HigDors
Accens the Laundretir.
Dec0ut-15-80
The Wfforsstiss that appears when clicking EUI), 530, 390, 590y la the same as tie informaaiinn dhat appeans whee clickteg (230, 300, 350, 510).
DecOut-15-88 + BigDecs
Access the Laundrette.
Claude
The umsge doennt show Interacthve eimecats or clickabte arros Rowrsee, at apoortmate comnitsates you mentiose (19), 130, 196, sidebor man the nelpuge. banon is part of servics offered FihurMeharge. wiich wolleds other options Fillens Cusppics Foss, Motor Repaic and scrcral
5501, grees buttow lnbeled "Eaheast"
Goblo
dotalliz counbattes (800
lokel correponn the Fomebation. infonmation Faushooid appeat and repairs.
Table 12: Qualitative test example for the task of GUI-VQA. GPT-4o (20240806), LLaVA-NeXT-7B, Phi3.5-V-4B and Qwen2-VI.-28 tend to hallucinate the response, mostly ignoring the input image. LLAVA-NeXT-7B and Claude are more conversational, however, the latter predicts correctly that the answer is 'Home Security'
Compunent
Cuntent
CCTV, Alarms and Security Services
Iaput Image
Query
Clickng 1680, 50, 750, 1100) gives noces to whut?
Gronn Truth
Display Home Security.
PhiX..V-48
'READ MORE"
P-4 BigDocs
Dippayy Hone Security.
LLaVA-NeXT-7
provided
de screstbett webste for that provides CCTV Alrmu, and Secarity Services. The
"Cleking (All, 50, 750, 110) the nocws whar?"
stoille the you oching about what clicking the "Koad
More" button would do, would licet sate the user pape with more detalld information about the services offered by mhe compasy. if
ie eaking aloas the contest the esage the visble parr of the tmage shawss secarity camera rounted on wall wiich is conntox piece of equpprad sand for somiillate secwely syriees.
LLANA-NeXT-BB BigDoos
Dippeyy the Home Secuitt.
Qwen2-VL..C 2B
Clickins (680, 30, 730, 179) glov the user sccess to Hone Autivnation.
VL.222-VL.21-- llggiees
Dupllyy the Home Security
DieOwl-15.88
The wer gvis oreen home uniomaaioo.
Di BigDecs
Dispayy Home Secuiity
Claude-3.5 Sonnct
Clicking area (680, 120) would giee access the 'Home Security acciion wetuite, be inferve coegaetic ofd
sarlgutios of page Rovw Theswestii*
olong
GPT-4 (21244886))
"Cliking would give
'Contact"
43

---

Published as a conference paper at ICLR 2025
Table 13: Qualitative example for the task GUI2Intent. All the models trained with BigDocs along with Qwen2-VL- 2ien2--V--2B produce the correct intent, while latter more conversational. Phi3.5-V-4B. LLaVA-NeXT-75 and DocOwl-1.5-8 infer incorrect sections of the image.
Compunent
Ingut Image
User
dlicks
view?
Genud Truth
Display HOME.
Phi35.V-4B
'Dopresmen" section the
PhiLS-V-4R- Riglbess
Display the HOME.
LLAVANNXT-2B
The vase yoa posvided appears be aveenstes of waloure with aerigamon the left sids the clicks the "Koutai""
menu Bem, which ia the diidd option from the dkey will lilely soles pape that pwoiid contert informatiio cosnac form for the
websts. This ien cvecooe froture websies bers sverz ras get mach wilk the onginiaation matn sdunk coomistt with the rive
LLXTA-NeXT-7B BigDoes
Duplay the HOME.
Owen2-VL..2 VL.2B
the saer click on the "HOME" link, they will be direcred to the main paze of the vebnite, nhich rppically connirs the homepage ora
sartgatton nenu wilk diferent sectooy of welvitt.
Q1.n2-VLL.BB BigDocs
Displey
Dee0wk-15.8
Whe riuks
(290, 1290,
60), dhy Tivthhrrpir blay.
Di BigDecs
Displyy
Table 14: Qualitative example for the task GUI2Intent. All the models trained with BigDocs along with Qwen2-VL produce the correct intent, while latter more comversational. Phi3.5-V.4B. LLaVA-NeXT-7B and DocOwl-1.5-8 provide confused responses.
Compument
hbf
VIC
Ingut Image
User
What
clicking 450,
Touth
P-135.V-BB
will
section
visblee
PHiXs.V-RR Righbors
Viee the About Us.
LLAVANOAT
The wase nestioned (800, 160, 450, 1901, the user would tiktt directed specific recios a clearer view of the valsits' lapont, o difirna devenine enerly who the sor sould cucomurr frowid te lué to sperle mevvr offordd the clists a page aboat particator dental procedurt, connee Sivs or an other tye of content that la relevant to cloic serviets.
prosided appsars is scromuoor of aueeuuee specifoalll dennd clinc's welsos By clickin the condinater of the welsiie. Howenes without wore conteat
LlaVA-NANT-11 BigTtox
View
QL.22-VLL2BB
wer directed the "Abowt section
webuite.
Qwen2-VL.2B BigDors
war will coconitr phone number cliblig regictte muster
HigDaes
44

---

Published as a conference paper at ICLR 2025
Table 15: Qualitative test example for the task GU12Sum. BigDocs models accurately capture the spatial orientation and stylistic axpects of of the webpage while making errors parsing the restaurant name. LLaVA- NeXT-7B expresses spatial disorientation, while Qwen2- VL-2B and DecOwl-115-8 provide too simplistic and incorrect responses respectively.
Component
Content
FOOD MENU
DRINK MENU
Input Image
User Query
you give coption the wolpaags image??
Ground Truth
layout TAPAS, DINNER, FEED ME, each with distinct color background and thar pricing The drnks nem, Ishelal "DRINK MENU!" is diulagal behow with vilmari doigs and inclales hest, wija, osckatis, The websile aloo inclakes social molin icom and lagin buikon, miaancng
This website
moden and colerail food
Genbeie.
offering Korean tupas small
erganzze, prominent "FOOD MENU" title
The divided
engagement.
PhiX5.
The imoge digdass a velpage with food and drisk wem for restguratt named Gunber. The up the page has as asvigtion aprions sack "Home', Gunbar Mono', ; "Oneer Online; "Lacation & Howry; BBook nve', and mecial medii icons for Instegmne and Facebook Them also Log In" option, The main content of the page is diidedd into noo wntionn: FOOD MENU and DRDIK MEND". The food mecu colorfat and lits variaay dickes with prices. inclulinn TAPSS" IMNEER; FLED MS. and "MALL PLAYES: tratoo memoon "WESFER"" GUUNBA FWLE, and "VEGAN aprion, The drid new partully visille showing WKEM and 'OO catggrtee une speofle beer brasdr and prices. The botten of image off and the drssk man not visibi..
Highooss
This fratares food vews fora
samed "Gustice" auto contest includes colorti organized with narkour categerized ""TAPAN"" O00,
webuite also includes clean eary novigare, beige barhguond averall
section variery food drink apalons.
destgned
LLAVA-NNN
The ohouee uelguge with food menu the left and drink and the drist menu daplays a selecian of beierages will their reapective prices. The design clean and organized, with color sthesee that includes shades of bluc. green, and orange. The Jeyar user, friendl, with cleur headings and easy real leat The overat impression that of mudern, autting exstamnn or caf swbstte.
right, The food men list variess stches wnth rivir
N6XT7- BigDoes
This webvite fiatures a abvvnt and modom mesa for remawrsst named "Rockin Rands, " "Te lasust is clean and organised, with beige baskgroand that counasy with the colectal mow ittss. The muis costent iacluies a ditatted food mesu with sections for bees cider, and drisks, rock listin nornies optione uill prices and deveejitins, froms and praphtiy. nuch hoer top and dreek, are uard vimuily repessent mon doms. wafsil includes nurigation options "Hom,," "Gestrabur Menn." "Onder Online," "Location Hours," icoss login Dumon, ovenall rtyle is conterporary and fisenll making essy for
olong onler
Qwen2-VL-28
"Cenuialy" Hew''s suption the wvbyagge: "Eqpenince
Gunbur
deliciour food and reforsing dred
Qwen2 Bighhes
This food page
foawrigg modow and clean
backyonud white
wahn
includes "FOOD MENU" sacrive with various catagarias like Tapus, Small Plate, Buffet, and Desserte, The leyost organiion with earigation har the upp large food
and 'DRINK MENU vecinn belons The color ahems primarlls durk with sccants green and siite creating a repiliticnttdd and briling smaretere. The uebitte also tnclueer social wedia looss and a login bumon for soets.
DieDul-15-68
"Fosd Menu" is the capiion for the wabyage the inage.
DieDul-L.5-9 5-988 RigDaes
This avbrir foouress a food menn for mossunn nooed "Gunbar." The legont rirss and orgasiged, with baige har barkgmand and a vil
bluch white. and oronge taat. The waln consont includes i denulled vecto with varlous calegorics rach ds appetizens, malks, and drinks, each with pricing and descriptions The nens Dres are steualt aypeatingg with bsages and froms. The website also har santgation oprionn for Houe, Gunbac Moon, Order Onlin, Location Hoars, Book now. Soctal mndia loou for Facebook Insitgrans are present the right.
45

---

Published as a conference paper at ICLR 2025
Table 16: Qualitative rest example for GUI2Sum. Phi3.5-V-4B and the BigDocs models are able to extract correct and relevant information from this simple and clean webpage, while the other baseline models make mistakes or are too simplistic in their caption.
Compunent
Cantent
Sorry. you have been blocked
Year andbe soous mww.cioouuuoccccccooooo
MEM (00 Mich RL1P
Imput Image
Query
Cus you give me caption die webpage the image?
Truth
website diplays "Surr,, vos bocked" eroor messuge, indicating
to acces rentaftcdeuttturcnooo
miniraaiinn biskked? "Whst caline snacks suggess delivering striighffewwal and, with
beckgmand this"" provided Cluudiaaee
symbhhbring
syinni
for assistacce. ovenil
informatio clearly.
V-48
indicting that the has bven blacked from accaczing the website rntrofeciondllars...o There a farge ref 'X' gubel is the conter, which epicolls represents block or enur. Bolow the 'X' there teo rectioss fest presiing rnforsation os thy the woer night fow been biscked and whut oclion can by token in mouler the bear. The not mugest dut the blok soull be due to zecartly sorviee protecing the webnte feve online anael and list potentual actions that could bave niggered the bisch, Ther slio a mustion of Cloufflare Kay ID and an addrers, along with a sote that mee performanc and security are by Civuglore
The ohonas aelpaay with
P- :lggloss
The uetuir deppays a bicceed access wessoge with a promieen red "XX" raubel The nuil costent expintss that the user has been binched niggerine ascurity moluion, pozsilly from sulminizz vertis words phrsues SQL commands malformed daia. The lapoat mininolists,
background sisple, derige. average pmnsidesn Couflarr Ray for
except for suppors
Closifoon logo
tach
LLAVA-NeXT-7
capiaan volyaue
could
Denied: Your Accoutt Blacked- Unable
Rentufirs.com" t0
N6X Bighnes
wobsie dippass
hax been blocked"
bidicoting
contottciodffeeesccoo The
exphitinn blcch dhe security service protecting the welvite from online sltncks and sugeen contacting the aiss with provied Cloudjlare Ruy for
sisiwalltt,,
background and central "X" symboliinng the Heck the
resolation, The ovenal sn clss and sniighffrradd focasing dolieeiigg the necessory inferatioon skarr:
Qwen2-VL.22 VL-2I
Sorry you have bees Biscked Unable to access voncofelooggurren.cmm
Qnon2-V1.28 HigDoes
The webnte deppegr blocked accass masegge with u prominsnt rnd "X" syrahal, The main contaat exphiis that the oor has besn Mlached dhe to miggoring a recurtty rolution possin shough actiny lire minimnngg certann words or phrases, SQt commands, meiforated data The keyoa aanimaticl,, with a white barkgownd and muple, ciean design The wu mnsighifonued and and informmtie, passidng mossur for the bluck and suggeting conneting ai owner resolation The Clonfaaae
included for further aurisunce..
DicDwl-15.8
The webyape du imuge deplors crez
versage Sorrs you hane bven blocked.
Di00wi-115-0 BigDess
The wchatte dispaus blocked pccess message with prominent triguasing
gimbol naie contror erplatu that the sser white imppmppriate from eutpp 'X" Clesslars logs loyar oener reselntion including Chouffaree for reference.
baskgmaan sieple, desigs. The
the
block and suggestr
46

---

Published as a conference paper at ICLR 2025
Table 17: Test Example for the task of Table2Latex. BigDoes models preserve structure of the table, lines from rows and columns, and font styles like bolds or italics.
Component
Content
dynamics( ACC)
Continaous dyuamics(or)
SiRt
SIS
Threshold
Kirman
Gone
Mutualistic
CML
T+1 1
0.85
0.86
0.89
0.54
0.588
0.958
0.017
7 T
0.73
0.80
0.84
0.81
0.602
1.086
0.021
7 +
0,81
0.75
0.81
0.82
0.609
1.276
0,024
4-4
0.74
0.83
Input Image
T 7
0.50
0.74
0.72
0.85
0.822
1.601
0,028
User Query
Please gencrate See necessary LaTeX sccije draw this table
Diseseee dynaniss (ACU)
50%
Thresbold
Kirman
Giome
Mitualieic
CML
3:27
0.305
U:30
3107
T
0.62
5.26
0
I8724
0107
0.41
3.28
1672
1
16922
1.911
01036
Thike Compurann defmmmn radkak dieret oortensora
dynamics(ACO) Threhold
Contimuou dynamirss(e)
SIR
SIS
Kirman
Gene
Mutuslistie
CML
0.85
11.66
0,80
0.84
0.308
0.568
0.017
7
1.086
+
0.81
0.75
0.81
0.82
0.609
1.270
0.024
4
0.74
0.83
0.724
1.512
0.027
PhE3.S-V-4D V-4B BigDoes
Disereto dynamics (ACC)
Contimnous dynamics [(]
Threshoki
Kirman
Mutualiste
CML
7 +
0.85
0.86
0.89
0.84
0.508
0.958
0.017
T
0.84
0.81
0.602
1.086
0.021
T
4
0.81
0.75
0.81
0.82
0.609
1.276
0.024
T 0.82
0.74
0.74
0.83
0.724
1.512
0.027
+-
0.72
0.822
1.601
0.028
Gp146
Table 18: Test Example for the task of Lable2 Phi3.5-V-4B able to format the table with all the rows, and used incorrect lines, and font style (boldness of first column row).
Composent
Chain
Chain
Chain
Chain
Chain
0.70
0.83
0.78
Chain
0.76
0.67
Chain
0.67
0.60
Chain
0.7d
Input Image
User Query
(Create LaTeX scipt illustrue this sable.]
Chain
Chain
Chain
Chain
Chain
0.70
0.61
0.67
0.76
Table 1: Table of Chains
PM
Chain 2
Chain 3
Chain 4
Chain 5
Chain
0.81
0.70
0.83
0.78
Chain
0.61
0.76
0.67
Chain
0.67
0.60
Chain
0.76
PM-3.5VV-4 BigDees
47

---

Published as a conference paper at ICLR 2025
Table 19: Test Example for the task of Table2Latex: Boldness and details of lines are perfectly aligned with input in the Phi3.5- 5-B--B BogDocs example.
Compoment
Content
Logic Gates
(0,0)
(0,1)/(1,0) (1.1)
AND
NAND
OR
0
NOR
XOR
0
XNOR
Tepul Imsage
Query
[Picner gaurnte necraaryy Lafex unije to sabic
Logic Gates
(0,0)
(0.1)/11,0)
(1,1)
AND
0
NAND
OR
0
1
NOR
0
XOR
1
XNOR
0
Phi3.5-V-48
Logic Gates
(0,0)
(0,1)/(14) (1,0)
(1,1)
AND
0
NAND
I
OR
0
NOR
1
0
XOR
0
0
XNOR
PhE3.5.V-4B BigDees
Logic Gates
(0,0)
(0.1)/(1,0)
(1,1)
AND
NAND
OR
0
NOR
XOR
0
XNOR
GPT40
Table 20: Test Example for the task of Table2Latex BigDocs failure. Phi3.5-V4BB-BiggDoc able
format question marks the first row.
Component
Mished
RGDN
PURE LET
DPB 2 20115
DWDX 22
PbD-Nec (Ours)
FANE
10.00
20,8
25.45
Input Image
SSM
0.426
0.501
055s
03561
0.568
Query
Pllase crooee LaTeX code produce this table.]
Method PSNR SSIM
RGDN 19.80
PURE-LET
DPIR
DWIDN
PhD-Net (Ours)
20.58
22.09
22.85
23.48
V-4B BigDocs
48

---

Published as a conference paper at ICLR 2025
Table 21: Qualitative test example I for the DeepForm task. When asked question that cannot be answered from the image, Phi3.5-V-4B + BigDocs(OCR reduction) hallucinates response while Phi3.5-V-4B + BigDocs correctly answers in the negative, demonstrating the role of OCR data in enhancing precise information extraction.
Compunent
AGREEMENT FORM FOR
NON-CANDEDATE /ISSUE ADVERTISEMENTS
Mattion and Laratiore
1entth
of Pschagg
Dbpt
Clsss
Week
Romber
Salidbii
8/21- 8/21-8127
Charges:
11, 325
Doea the programmiing (In whole or parta contrnstiins Va
Input Innage
Query
the contoct.uuss
Phi..5-V-4B Bightss
Nane
PHi.5. V.48
BighhesOCR roduction)
12-15
49

---

Published as a conference paper at ICLR 2025
Table 22: Qualitative test example 2 for the DeepForm task. When asked a question that cannot be answered from the image, Phi3.5-V-4B + Big (OCR reduction) hallucinates an unrelated response from a different section of the image while Phi3.5-V-4B BigDocs correctly answers in the negative.
Composoul
Contcol
FEDERAL STAERALSSATION ISSUE ADVERTISING REQUEST SHEET
(Io (llddes and lags publs (mgestan))
Thi regpost
Pres:
Nediocll Modii Grep
Dake
2013
Ageec
Nedioal Media Creg
51, MT 1801
Plc
212-434-110)
Caakaat
Candider Murione
Flesion Werlbook Crog
Addrec: Time SE
Wt.l Wulriiuu UC
Theae 360-479-000
THin Eureatiee Dricee
Image
Use Query
What the value for the fight
Ground Truth
None
V-43.5--44 Bighhes
P-43..V-4B BiR
National Media
Table 23: Qualitative test example I for the InfoVQA task. Phi3.5-V-4B + BigDocs is able to use the combination of the percentage and the pictorial representation of the relevant data and produce the correct count while Phi3.5- HigDocs(caption reductionn gets confused with a different region the image, demonstrating the robustness text and image representations that captioning data brings.
Companest
THIS ISN'T JUSTICE.
There are 832% MORE
women in prison now than in 1977
Black incarcerated
3 TIMES MORE than white women
1 IN 25 o will give birth SBACKLED peison.
75 %
ARE DOMESTIC VIOLENCE SURVIVORS
IN 10 will be sexually assaulted in prison
are survivors of 82% SEVERE CHILD ABUSE
color of Wehanggeorg 1t Ttrraa iolet
SHARE YOU THNK IT'S TMEE TO STOP CAMINALIZN WOMER AND SUFVIVORS
Image
Use Query
Bow ransy wormnn rur of every 4 wemen are domacic vusienee surVANes
Ground
Phi05-Y-48 Bilbes
PhB.5.V-4B
RiDDosicaptinn coluction)
50

---

Published as a conference paper at ICLR 2025
Table 24: Qualitative test example for the InfoVQA task. When tasked with extracting a percentage based on textual and pictographic depiction of data, Phi3.5-V DigDoss(caption reduction) incorrectly picks percentuge value in the vicinity, while Phi3.5-V-4B + BigDocs correctly infers the answer.
Conguoee
Content
CATI
AMERICAN
THE AMERICAN IDENTITY
AUCISCAN
ARR BCCAS
ACCORDING O SOCIAL MEDIA
AMGRKASS
MOCRLEDE
JUNE SAMES
ABC CESISEE
AMERICANS ARE SOCIAL
Use Queey
What peroonage of Ameicaas are otline!
Grunnd
Phi.5---48 Righasicaption reduction)
70%
51

---

Published as a conference paper at ICLR 2025
Table Qualitative test example for InfoVQA task. Phi3.5-V-4B BigDDos is able to exploit structure in the image and compute the correct answer, while Phi3.5-V-4B + RigDocs(captinn reduction) hallucinates an unrelated answer.
Comguneet
Content
CIVIL WAR FACTS
1861 -1865
EDETERCI PESTICIIT SOSE
DORTER STORS
ROMEST:MM
FINANCE Deporit
$234 million
$74 CORF.SEAECT
BEASAD STAIES
POPULATIONS
BORDER
outeed
AGRICULTURE
With scoeption oftor cat trowon Popoclach buctec Be taan lad reverdand
O1STEXCRACE
Use Query
How long (ie yean) was the Anss xon Civil Wa.?
Grund
Ph0-5-Y-48-
PhiLS---48 Righesicaption reduction)
52

---

Published as a conference paper at ICLR 2025
Table 26: Qualitative test example for the InfoVQA task. Phi3.5-V-4B BigDocs(no VQA format) does not
adhere to the requested format and also provides wrong answer in contrast to Phi3.5-V-4B + BigDocs which gets both the format and the answer correct. These errors highlight the importance of VQA formatting of the BigDocs dataset.
LEARN THE FACTS ABOUT SEXUAL VIOLENCE
WOMEN
WOMEN
Mereiia and
26.2%
EJ
MEN
675% INSTANCES ESTIMATED UNREPORTED."
OUT OF
SEXUALL VIOLENCE WHEN TAKEN SERIOUISLY VICTIM BLAMING GOES UNCHECKED.
USE YOUR VOICE TO PREVENT IT WELLEVE BURNNRRS WOMEEME ELTMELA CRESPET BORAIREES
S NSVBC KIAAGU
Input Innage
Use Query
What percentage of wommn www.://www....oooooooooooooeeeee attrrrrd
Phi Bighttes
20%
Phi..-V--t-- Hightsstoo VQA fummn))
J svam hev experirnced serual stolerr other shan divir ljpttos.
53

---

Published as a conference paper at ICLR 2025
Table 27: Evaluation example from the Image2HTML task. The HTML code is rendered for easier comparison and analysis. The original Phi3.5-Vision performs poorly, while the BigDocs fine-tuned version generates highly accurate results. Although GPT-4 produces correct its structure differs significantly from the original image. These results demonstrate that fine-tuning on BigDocs leads to a massive improvement in performance on this task.
Componcnt
Ditgiaal
Phit Outpur (Readered)
Choppe
GITT Dalput (Randend)
Table 28: Test Example for the task of GUI-VQA, showing BigDocs model failure. Phi3.5-V44B+ was not uble to give the correct output.
Component
X
OPANICA
SPAN
Iupat looage
Query
[1 clicks (400, 780g will they vicv?]
Ground
Display
+ BigBocs
Display DRZAVA.
54

---

Published as a conference paper at ICLR 2025
Dutrbct Blans Cruad
Deinisaton Toor in Roder Ouloza
Figure 18: Comparison of token length histograms on the test set of Screenshot2HTMI..o task, and the generated outputs of our model (Phi3.5-Vision+ Vision+ BigDoos)
1ilee vich 19M0 wanm TESLUm
suaer woreet astiraad separe quee. tagpee,,
Figure 19: Failure case the task of Screenshos2HTML Output is generated by the Phi3.5 Vision + BigDocs model.
Context Length
HTML Score
HTML Score
Gready
Longh penalty --1
512
10,99
13.96
1,024
11.49
15.80
2,048
9,81
14.71
4,096
8,83
13.88
8,192
8.38
13.60
16,384
8.24
13.43
Table 29: HTML Scores for Viii.--Visionn-BBDDDos with varying context lengths. HTML Score defined as DOM Tree Edit Distance.
55

---

Published as a conference paper at ICLR 2025
ScRReCTIE teat PUBLEC MC/ORE//OTO WITN, 3:E Topeectionn//EE "EIERESAAANNN ong www.araaeoeeeetrrrrt..o.oo. www.e//twswoeteee.......
-aare contest centest "tect/htel; charoet-atf-2" trtp-egstt "aapeetttt"""""tttt"pp"""""
gite stete's Sccretary of Hery Selectly:: fraubilent, stogarce, or sntefernedf
<choah
sioth: LibRoa; reighti 128x4"">
[tect]
Hereary and veccones
Paat-ngter stace's a Secretary of leeits Mary Selectiy; traguiant, aiequrted, or satifermed?
by Teresa Stwescck
Seg 39, 2299
Bree apuir, hishly placed pablic rigure has presented misiaacing or inacorrnce statenent wherete intrersmt is orreresaly decribed is namiase.
3: anticte about thineroset in suow fi's shots, reperter fandi teaghtzen's a sumorized Nashington tealeh Department's Secretary of festts Mary Selectky is texing said, "Th
13 Mary Solectly slaying vort pance tret differentsate actrece noults probices and, in contreat, tharcrouct indkee dess coppentts colayst ia the rst overs pcen revooe fonk
Corciuaters in seversl stuties prorst coecem (e) for fandi tasgicer's a reporting of that Mary Selectly sete, or cal for Mary Selectly sate.
Corsuder secerel stucess Nele infants ahs recelsed thaerooeloomteent tout seciination vere rise thes is likely to te recetsirg special scceatier services (D) and
Keetione: Sellctlo sosupted 1f
thiremaatis adverse effects and accurataly sustec thra walts atfecte", Pary
</2-
Maagerse"
thinersual's cafery fieclage cuggect guite the oppacine) relessa quectinnc iicluee, profice
SETTEEE redtue SEATTLE CASTTLETTEEEE BESEEEEssnssuss Sew.EEETTTTTSSSSSe cwntant-SSAAT SEATE--SSTTTTTEEEEE Err.EEEESSSam.... melandde eTpPEES
terppessu centon LEAATTL EETTELETIMMMAN Monooaa
tule conteet AOSATI Prieesampss wnd cowaeaaTTEE SOOEE Hhe StATT
with CRNDUIEEN CONIEESAAATT LET Ien confo TeseETTTEE ute DeSSSaa WOGNTT
Figure 20: Failure case on the task of Screenshof2HML Ourput is generated by the Phi3.5 Vision + BigDoos model.
MIBOCTTT bost PUBLEC MNODONCCOOT BITHL 3-8 Trarectiona//11"" "HIEE://WNN stylen
untng wwwpeeeeeooeeeeeeeatttttooo
<mere content "tect/hrets cherastratt-g" ttpp:estt=""tattee--e""""""
Secretary Horlth Mery Selectly: frawkllstt sispacel, or sninferre!?
.cheath "werflaw tiidee stett: Lisees; heigar: "aax"""
[bect]
Bereary and wecconss
Mask-ngter stace's Secretary of Health Mery Sctectiy: traguiest, atoquoted, or intnfermed!
by Teros Bustock
See 38, 2097
free apair, hiphty placed paotic figure has presented misleating or inacourate statenent whereie Uninerosot is orreroxslty descrited is namiess.
3r enticte ebout chanerosel in soire thes stors, reperter taeli bragition's semorkzed Nookington toulth Deuertentts Souretary of Moatt Mary Selectky as hering said, "The
15 Mary Selecite stouing wont goess thet eifferentoae setuece nealth probtces znt, in contrest, tharcrose undieed dese cxpennttt delayst is she not overt peorirevione tor
Corclus clsiters in seversl stotles prorst coecer (00 for tandi Saughcen's i reporting of weat Mayy Solectly soad, or (l) for Mary Selectiy sate.
Jee secorel stains, tele infanta aes roccianl it thaeoosel contiiiiiin towe wuth ees tiet bye, vare riee Lhes is likely to le recpining iocial rostti serviees It) usl shanerost
other
busttons: No Sedectly ausoutedf 26 est, w/1-
thdremanti adverse effects and accuratety zuster ttre waits offects",, ane iirry
recarnaers
cafecy flecdage cuggert oppacire)
ixclude, profice
Vie
SeSee SOAASSAATLLE TIL
MedEATTLLTTNSSeeee contastesSATIL SASTSAATTTTTEDE TELEEEESSaansss motianns
tarrt CAATTT THE aetouiu
IotieEtETTTEEE... Mo st Soppena TL selteettie soon ala aaSERTT teeie pin cappoi con Le
restee seat Kiseooa and
ata EATT Comp
DISESNNII COME WAR ILE
KUMG ahe www.erowmeoooooeeeeeeeeeeo
Figure 21: Failure case on the task of ScreenshorHHMMI Output is generated by the Ph Vision + BigDocs model.
56

---

Published as a conference paper at ICLR 2025
<iile
vith Dab Sipe""
with,
dedden
Testing koudee anseest ou 1907, damrts vsa Praatets tole mutl Loddee
wajerss tan
SHORE Street
"Munk leser Sume?
Mveer Wil Rovry surtan an mepte steeds
Figure 22: Success case on the task of Screenshor2HTML Output is generated by the Phi3.5 Vision + Bic model.
57


Reviews:
[{"review_id": "mFzPoS3uyY", "reviewer": "Reviewer_gUaq", "summary": "The paper introduces BigDocs-7.5M, a large-scale, license-permissive dataset for training multimodal models on document and code-related tasks. Along with a comprehensive suite of tools and data analysis, the authors present BigDocs-Bench, featuring 10 downstream tasks that assess a model’s ability to generate long-format code outputs from images. These tasks serve as practical benchmarks for real-world applications. In paper the experiments show that models trained on BigDocs outperform those trained on existing datasets. All the artifacts in this paper is open source and permissive.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. All the stuff in the paper is under permissive license, which is a big plus for an artifact and benchmark focused research paper.\n2. The dataset curation process and filtering make sense for better data quality\n3. Bonus point on keeping multimodal in mind when creating a document-understanding dataset.\n4. Data contamination analysis is nice to have when the dataset proposes training set.", "weaknesses": "1. Many aspects of the data curation process, including the effect of OCR, VQA format and text-image alignment, etc. are sort of still unanswered in the experiment section, making readers wonder, why and if these curation processes are needed, or are they actually contributing to better performance resulted in the final model.\n2. As a aggregation benchmark with a lot of different downstream tasks, how are the aggregated score on this BigDocs-Bench correlates with other existing benchmarks also presented in the paper? A comparison of what new aspects this new Benchmark is adding to existing research landscape is crucial to justify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdMXwTrl4I", "reviewer": "Reviewer_UxFn", "summary": "The paper introduces BigDocs-7.5M, a large-scale, open-access dataset (CC-BY-4.0) designed to enhance multimodal model training on document and code-related tasks. Addressing limitations in existing datasets such as restrictive licensing and data access, BigDocs-7.5M provides a rich collection of 7.5 million multimodal documents suitable for a variety of tasks including document reasoning, structured output generation, and graphical interface interpretation. The authors alrso release the BigDocs-Bench to evaluate LLMs ability to analyze code and docs over 10 categories of tasks.", "questions": "While I appreciate the authors pro-actively acknowledging that the limited context length of the models trained (8192 tokens) might impact the performance of the models, can the authors provide some insights on the artifacts of this decision. For example, we can see both the 7b and the phi-3.5 models saturating around 50%. Do the authors suppose this might be an artifact of the context length? On the same lines, or perhaps, is there another reason for the plateau? Or is it merely the dataset getting harder beyond half the samples?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Comprehensive and Novel Dataset: The introduction of BigDocs-7.5M offers a permissively licensed, open-source dataset that includes a wide variety of document types and structured outputs\n\n2.  Improvement in Model Performance: The paper convincingly demonstrates that models trained on BigDocs-7.5M outperform those trained on existing closed-source datasets, particularly in multimodal document understanding and code generation tasks​ with Phi-3.5 Finetuned at ~50% vis-a-vis gpt-4o ~25%. \n\n3. Thorough evaluation suite: The introduction of BigDocs-Bench for benchmarking model performance across 10 novel tasks is a valuable contribution, providing detailed insights into the models' capabilities.", "weaknesses": "1.With regards to human-evaluation in section 5.3 of the paper, could the authors shed more light on the evaluators' qualifications, selection process, and any potential conflicts of interest? \n\n2. While I do appreciate the value of BigDocs-Bench, and the evaluation on a collection of open-source and closed-source models, I would encourage the authors to consider including a correlation analysis between BigDocs-Bench and specific benchmarks like human-eval and RULER. This would provide a clearer picture of how BigDocs-Bench relates to existing evaluation metrics in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjUoc546mg", "reviewer": "Reviewer_1wn6", "summary": "The paper introduces a new large-scale and license-permissive dataset BigDocs for continual pertaining on multimodal document and code tasks. Based on this dataset, the paper also introduces BigDocs-Bench containing the training set, validation set and test set to evaluate tasks like code generation and reasoning from documents.", "questions": "+ 'BigDocs will be open-sourced (upon acceptance)'. If you want to open-source this dataset, what's the point of waiting upon acceptance?\n+ Personally, I think it's not necessary to keep a *hidden test set* for the benchmark.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper introduces a new large-scale dataset for pertaining and fine-tuning, and a corresponding benchmark, with permissive license, which is the first one in the target domain. This is a concrete contribution to the research community and industry in the subarea.\n+ The paper presents comprehensive experiments with leading open and closed models on both general and proposed benchmarks. The results are promising. Also, human evaluations provide further evidence on the effectiveness of training on BigDocs.", "weaknesses": "+ The results on the proposed BigDocs-Bench are a bit strange. The performances of the same model on different tasks are not stable, e.g., Claude-3.5-sonnet behaves very bad on Chart2MD and gets a high score on GUI2Sum. Further, small models can gets higher scores than a much larger model, e.g., Qwen2-VL-2B gets a higher avg. score(20.00) than Claude-3.5-Sonnet(18.31) and GeminiPro-1.5(19.23). This could indicate the benchmark is too specific for a general model not trained with BigDocs.\n+ In human evaluation, gpt-4 gets a higher win rate than Phi3.5-BigDocs, while in BigDocs-Bench it's the opposite, which again indicates the benchmark may not differentiate a stronger model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgDKBMxCOX", "reviewer": "Reviewer_ZUCq", "summary": "This paper introduces BigDocs, an open-source large-scale multimodal dataset for document understanding and code generation tasks. The main contributions include:\n\n1. BigDocs-7.5M: A high-quality dataset containing 7.5 million image-text pairs across 30 tasks\n2. BigDocs-Bench: 10 new benchmark tasks focusing on structured output generation\n3. BigDocs Toolkit: A set of tools for data processing and preparation\n4. Experimental validation: Demonstrated effectiveness through comparisons with models like GPT-4", "questions": "In addition to the three points mentioned in the weaknesses section, I am concerned about the performance reported in Table 3. The unusually low performance of advanced models like GPT-4o and Claude-3.5-sonnet raises questions about the experiment's setup. Did you use one-shot prompting in your experiments? If zero-shot prompting was used, this might create an unfair comparison with models that haven't been fine-tuned on these specific benchmarks. Could you clarify the prompting strategy and consider providing results with one-shot or few-shot prompting for a more equitable comparison?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Addresses a significant issue: licensing restrictions and accessibility problems in existing document understanding datasets\n\n2. Quality assurance:\n        Rigorous data filtering process\n        Traceable metadata\n\n3. Comprehensive task coverage: from basic document information extraction to complex structured output generation\n\n4. Open-source commitment: supports responsible AI development", "weaknesses": "1. Insufficient validation of benchmark quality and reliability\n\n    a). While Section 4.2 mentions manual human verification for BigDocs-Bench, the paper lacks crucial details about the verification methodology and evaluation criteria. For example, the number of human verifiers involved, their qualifications, the specific criteria they used for evaluation, or how inter-rater reliability was ensured.\n\n    b). Given the large volume of synthetic data in the benchmark, the paper fails to address the practical challenges of comprehensive human verification. The sampling strategy and quality assurance process for human verification are not described, raising questions about the robustness of the validation process\n\n2. Limited scope of base model experiments\n\n    a). The experimental validation is confined to models ranging from 2B to 7B parameters. The absence of experiments with larger-scale base models (>7B parameters) limits the understanding of the dataset's effectiveness across different model scales\n\n3. Insufficient qualitative analysis of model performance\n\n    a). The paper lacks detailed error analysis and concrete examples in the Qualitative Results section.\n\n    b). Section 5.3 would benefit from including error distribution patterns across different models and tasks. The absence of systematic error analysis makes it challenging for readers to:\n\n        i. Understand the specific improvements BigDocs-7.5M brings to different document processing tasks\n\n        ii. Identify potential limitations or biases in the dataset\n\n        iii. Comprehend the typical failure modes of models trained on this dataset", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)