KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding
Authors: Ahmed Heakl, Muhammad Abdullah Sohail, Mukul Ranjan, Rania Elbadry, Ghazi Shazan Ahmad, Mohamed El-Geish, Omar Maher, Zhiqiang Shen, Fahad Shahbaz Khan, Salman Khan
Accepted at: acl25

KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding

Ahmed Heakl♣♣*, Abdullah Sohail♣♣*, Mukul Ranjan♣♣*, Rania Hossam♣♣
Ghazi Ahmad♣, Mohamed El-Geish♣, Omar Maher♣, Zhiqiang Shen♣
Fahad Khan◊◇, Salman Khan♣◇
♣ MBZUAI   ♦Monta AI   ◊Linköping University   ◇Australian National University
{ahmed.heakl,mabdullah.sohail,mukul.ranjan,salman.khan}@mbzuai.ac.ae
{geish,omar}@monta.ai
https://mbzuai-oryx.github.io/KITAB-Bench/

Abstract

With the growing adoption of Retrieval-Augmented Generation (RAG) in document processing, robust text recognition has become increasingly critical for knowledge extraction. While OCR (Optical Character Recognition) for English and other languages benefits from large datasets and well-established benchmarks, Arabic OCR faces unique challenges due to its cursive script, right-to-left text flow, and complex typographic and calligraphic features. We present KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in current evaluation systems. Our benchmark comprises 8,809 samples across 9 major domains and 36 sub-domains, encompassing diverse document types including handwritten text, structured tables, and specialized coverage of 21 chart types for business intelligence. Our findings show that modern vision-language models (such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like EasyOCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate (CER). Furthermore, we highlight significant limitations of current Arabic OCR models, particularly in PDF-to-Markdown conversion, where the best model Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in accurately recognizing Arabic text, including issues with complex fonts, numeral recognition errors, word elongation, and table structure detection. This work establishes a rigorous evaluation framework that can drive improvements in Arabic document analysis methods and bridge the performance gap with English OCR technologies.

1 Introduction

With the upsurge in adoption of Retrieval-Augmented Generation (RAG) based systems for document processing, the quality of document ingestion pipelines has become increasingly critical.

Figure 1: Overview of the core domains and sub-domains in KITAB-Bench. Our benchmark spans nine major domains (e.g., OCR, charts to JSON, table recognition) and 36 sub-domains (e.g., scanned text, handwritten text, various chart types), providing a comprehensive evaluation framework for modern Arabic document processing and analysis.

Optical Character Recognition (OCR) plays a crucial role in this pipeline, enabling the conversion of physical documents into machine-readable text and databases for enabling effective knowledge retrieval. Although significant progress has been made in the multilingual OCR (JaidedAI, 2020; Fu et al., 2024; Wei et al., 2024; Smith, 2007), with comprehensive datasets like PubLayNet (Zhong et al., 2019b), DocBank (Li et al., 2020), M6Doc (Cheng et al., 2023), and DocLayNet (Pfitzmann et al., 2022), Arabic OCR continues to lag behind. This gap is largely due to the unique challenges of the Arabic script, including its cursive nature, complex typography, and right-to-left text orientation.

Existing Arabic OCR datasets (Table 1), like KHATT (Mahmoud et al., 2014) and IFN/ENIT (Pechwitz et al., 2002) focus mainly on handwritten text, whereas APTI (Slimane et al.,
Figure 2: Overview of different tasks in our benchmark: Eight key components illustrating the task inputs and outputs for table recognition, chart understanding, text recognition, diagram analysis, VQA, line detection, layout analysis, and PDF-to-Markdown conversion, complete with input/output examples for each task.

<table>
  <tr>
    <th>Domain/Characteristics</th>
    <th>EXAMS-V*</th>
    <th>Camel-Bench</th>
    <th>MIDAD†</th>
    <th>KHATT</th>
    <th>KITAB-Bench (Ours)</th>
  </tr>
  <tr>
    <td>PDF to Markdown</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Layout Detection</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Line Detection</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Line Recognition</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Table Recognition</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Text</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Charts to JSON</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Diagram to Code</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>VQA</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Handwritten Samples</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Open Source</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>Total Samples (#)</td>
    <td>823</td>
    <td>3,004</td>
    <td>29,435</td>
    <td>5,000</td>
    <td>8,809</td>
  </tr>
</table>

Table 1: Comparison of Arabic OCR Benchmarks Across Different Domains. Benchmarks compared: LaraBench (Abdelali et al., 2023), CamelBench (Ghaboura et al., 2024), MIDAD (Bhatia et al., 2024), KHATT (Mahmoud et al., 2014), and KITAB-Bench (Ours). (*: Only the Arabic samples are considered.) (†: The test set of the dataset is considered.)

2009) covers only specific aspects of printed text. These efforts fail to address advanced document processing challenges such as table parsing, font detection, and numeral recognition. Arabic benchmarks like CAMEL-Bench (Ghaboura et al., 2024) and LAraBench (Abdelali et al., 2023) evaluate large multimodal and language models, but they give limited attention to document understanding tasks. Consequently, there remains a need for a more comprehensive framework to systematically evaluate and compare Arabic OCR solutions. Our benchmark addresses these gaps by offering diverse document types and evaluation tasks to facilitate in-depth assessments of modern OCR systems.

We present KITAB-Bench, a comprehensive Arabic OCR benchmark spanning 9 domains and 36 sub-domains. Our framework evaluates layout detection (text blocks, tables, figures), multi-format recognition (printed/handwritten text, charts, diagrams), and structured output generation (HTML tables, DataFrame charts, markdown). This enables rigorous assessment of both basic OCR capabilities and advanced document understanding tasks.

The contributions of this work include (1) A comprehensive Arabic OCR benchmark covering multiple document types and recognition tasks. (2) Detailed evaluation metrics for assessing performance across different document understanding challenges. We also propose CharTeX and CODM metric to evaluate chart extraction and diagram extraction respectively. (3) Baseline results for popular OCR systems and Vision Language Models (VLMs), highlighting current limitations and areas for improvement. (4) A standardized framework for comparing Arabic OCR systems, facilitating future research and development.

2 Related Work

The development of robust Optical Character Recognition (OCR) systems has been extensively studied across document layout analysis (Zhao et al., 2024; Shen et al., 2021; Paruchuri, 2024b; JaidedAI, 2020; Auer et al., 2024; Li et al., 2020), table detection (Li et al., 2019; Paliwal et al., 2019; Nassar et al., 2022; Li et al., 2021; Schreiber et al., 2017), and document understanding (Staar et al., 2018; Weber et al., 2023; Livathinos et al., 2021). While English OCR benefits from rich
Figure 3: Comparison of model performance across four document understanding tasks (Table Recognition, Image to Text, Diagram to JSON, and Layout Detection) showing successful and failed cases for different models including Ground Truth, EasyOCR, GPT-4, Qwen, Surya, Tesseract, Yolo, and DETR on Arabic document benchmark data.

<table>
  <tr>
    <th>Domain</th>
    <th>Total Samples</th>
  </tr>
  <tr>
    <td>PDF to Markdown</td>
    <td>33</td>
  </tr>
  <tr>
    <td>Layout</td>
    <td>2,100</td>
  </tr>
  <tr>
    <td>Line Detection</td>
    <td>378</td>
  </tr>
  <tr>
    <td>Line Recognition</td>
    <td>378</td>
  </tr>
  <tr>
    <td>Table Recognition</td>
    <td>456</td>
  </tr>
  <tr>
    <td>Image to Text</td>
    <td>3,760</td>
  </tr>
  <tr>
    <td>Charts to DataFrame</td>
    <td>576</td>
  </tr>
  <tr>
    <td>Diagram to Json</td>
    <td>226</td>
  </tr>
  <tr>
    <td>VQA</td>
    <td>902</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>8,809</th>
  </tr>
</table>

Table 2: Distribution of samples across different domains in our dataset. A more detailed count for different sub-domains and data sources is in Appendix A.

datasets like PubLayNet (Zhong et al., 2019b), DocBank (Li et al., 2020), M6Doc (Cheng et al., 2023), and DocLayNet (Pfitzmann et al., 2022), Arabic lacks standardized benchmarks for diverse fonts and layouts. Recent efforts like MIDAD (Bhatia et al., 2024) curates extensive training data for Arabic OCR and handwriting recognition, while Peacock (Alwajih et al., 2024) introduces culturally-aware Arabic multimodal models. Existing resources such as CAMEL-Bench (Ghaboura et al., 2024), LAraBench (Abdelali et al., 2023), MADAR (Bouamor et al., 2018), OSACT (Mubarak et al., 2022), and Tashkeela (Zerrouki and Balla, 2017) focus on language modeling or specific tasks rather than full-page OCR evaluation. Handwriting datasets including HistoryAr (Pantke et al., 2014), IFN/ENIT (Pechwitz et al., 2002), KHATT (Mahmoud et al., 2014), APTI (Slimane et al., 2009), and Muharaf (Saeed et al., 2024) emphasize word/line recognition over document structure analysis.
Arabic table recognition faces challenges from merged cells and RTL formatting (Pantke et al., 2014). While methods like GTE (Zheng et al., 2021), GFTE (Li et al., 2021), CascadeTabNet (Prasad et al., 2020), TableNet (Paliwal et al., 2019), and TableFormer (Nassar et al., 2022) advance Latin table detection, their effectiveness on Arabic documents remains unproven. Document conversion pipelines (CCS (Staar et al., 2018), Tesseract (Smith, 2007), Docling (Auer et al., 2024), Surya (Paruchuri, 2024b), Marker (Paruchuri, 2024a), MinerU (Wang et al., 2024a), PaddleOCR (Du et al., 2020)) lack Arabic-specific optimizations for segmentation and diacritic handling (Mahmoud et al., 2018; Kiessling et al., 2019). This highlights the critical need for comprehensive Arabic OCR benchmarks addressing text recognition, table detection, and layout parsing.

3 KITAB-Bench

Our methodology offers a novel approach to benchmarking Arabic OCR systems via a comprehensive data collection strategy and a systematic evaluation framework. We gather curated samples from existing Arabic document datasets, manually collected and annotated PDFs, and employ a five-phase LLM-assisted human-in-the-loop pipeline (Figure 4) to generate diverse supplementary content. Our evaluation framework spans nine specialized tasks, enabling thorough assessment of OCR performance across various document processing challenges and providing a robust benchmark for Arabic document understanding tasks.

3.1 PDF Data Collection

We curated 33 diverse PDFs from online sources in academia, medicine, law, and literature. To ensure challenging cases, we selected documents featuring richly formatted tables with extensive color usage, merged cells, Arabic numerals, historical texts, watermarks, and handwritten annotations. Each PDF averaged three pages, and we then manually annotated them. This dataset comprehensively captures real-world complexities, making it a valuable benchmark for PDF-to-Markdown conversion.
Figure 4: Synthetic Data Generation Pipeline: A 5-stage process using LLMs to generate topics, create raw data, produce visualization code, render charts, and perform human evaluation for quality control.

3.2 LLM-Assisted Data Generation Pipeline

To generate data for charts, diagrams and tables, we implemented a five-phase LLM-assisted generation pipeline with human validation at critical stages, as illustrated in Figure 4. In Phase I (Topic Generation), our system employs an LLM to generate diverse topic names across multiple domains. This phase incorporates various personas (academic, legal, medical, technical) to ensure broad coverage of document types. Phase II (Data Generation) transforms the validated topics into structured raw data. The LLM generates content following Arabic linguistic and formatting conventions across various domains. In Phase III (Code Generation), the system converts the validated raw data into plotting code, with special attention to Arabic text rendering requirements and RTL content management. Phase IV (Image Rendering) utilizes specialized rendering engines (Mermaid, Plotly, Vegalite, HTML) to create visual representations while maintaining Arabic text integrity.

The final phase (Human Evaluation) implements rigorous quality control through expert validation. Evaluators filter charts, tables and diagrams based on detected anomalies and ensure adherence to Arabic-specific document conventions. This phase is crucial for maintaining the high quality of our benchmark dataset.

3.3 Dataset Statistics

Our benchmark dataset comprises over 8,809 samples across 9 major domains and 36 sub-domains, representing a comprehensive collection of Arabic document types for OCR evaluation. As detailed in Table 8, the dataset combines carefully curated samples from established datasets, manually annotation PDFs, and synthetically generated content created through our LLM-assisted pipeline (Figure 4). The Image-to-Text portion (3,760 samples) includes data from historical documents (HistoryAr (Pantke et al., 2014)), handwritten text collections (Khatt (Mahmoud et al., 2014), ADAB (Boubaker et al., 2021), Muharaf (Saeed et al., 2024)), and scene text (EvAREST (Hassan et al., 2021)), while layout detection comprises 2,100 samples from BCE-Arabic-v1 (Saad et al., 2016) and DocLayNet (Pfitzmann et al., 2022).

For layout analysis, we incorporated 1,700 samples from BCE-Arabic-v1 dataset (Saad et al., 2016), 400 samples from DocLayNet dataset (Pfitzmann et al., 2022) focusing on financial, academic, legal, and patent documents. The line detection and recognition tasks contains 378 samples each from self-developed dataset. We further enriched the dataset with 500 samples from PATS-A01 (El-Muhtaseb, 2010) benchmark to ensure diverse representation. For handwritten text recognition, we assembled a comprehensive collection of 1,000
<table>
  <tr>
    <th>Task</th>
    <th>Metric</th>
    <th>Surya</th>
    <th>Tesseract</th>
    <th>EasyOCR</th>
  </tr>
  <tr>
    <td rowspan="2">Detection</td>
    <td>mAP@50</td>
    <td><b>79.67</b></td>
    <td>46.39</td>
    <td>68.02</td>
  </tr>
  <tr>
    <td>mAP@0.5:0.95</td>
    <td>27.40</td>
    <td>14.30</td>
    <td><b>32.74</b></td>
  </tr>
  <tr>
    <td rowspan="2">Recognition</td>
    <td>WER</td>
    <td>1.01</td>
    <td>1.00</td>
    <td><b>0.53</b></td>
  </tr>
  <tr>
    <td>CER</td>
    <td>0.87</td>
    <td>0.66</td>
    <td><b>0.20</b></td>
  </tr>
</table>

Table 3: Performance of different models on Line Detection and Line Recognition Task on our Benchmark

samples combining datasets from Khatt (Mahmoud et al., 2014) (both paragraph and line-level annotations), Adab (Boubaker et al., 2021), Muharaf (Saeed et al., 2024), and OnlineKhatt (Mahmoud et al., 2018). The benchmark also includes specialized content from ISI-PPT (Wu and Natarajan, 2017) (500 samples), and Hindawi (Elfilali, 2023) (200 samples) for various document types. Scene text understanding is supported by 800 samples from EvArest (Hassan et al., 2021), providing real-world context diversity. A detailed table showing all the dataset is provided in the Appendix A.

A significant portion of our dataset consists of synthetically generated content, including 576 samples for Charts-to-DataFrame (spanning 16 different chart types), 422 samples for Diagram-to-Code (covering sequence diagrams, flowcharts, and tree maps), 456 samples for Tables-to-CSV/HTML, and 902 samples for VQA tasks. These synthetic samples were generated through our five-phase LLM-assisted human-in-the-loop pipeline (Figure 4). Every sample in our dataset - whether from existing sources or newly generated - underwent validation by native Arabic speakers before inclusion in the final benchmark. This rigorous validation, reinforced by expert review and automated checks, ensures high quality and authenticity across all domains. A detailed analysis is in Appendix C.

4 Experiments

Our experimental evaluation comprehensively assesses the capabilities of current OCR systems and state-of-the-art vision-language models (VLMs) across different Arabic and multilingual document understanding tasks. Figure 2 illustrates the nine distinct tasks in our evaluation framework.

We evaluate three categories of systems: VLMs, traditional OCR systems, and specialized document processing tools. For VLMs, we include both closed-source models like gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 (Hurst et al., 2024; Achiam et al., 2023), and gemini-2.0-flash (Georgiev et al., 2024; Google DeepMind, 2025), as well as open-source alternatives such as Qwen2-VL-7B (Wang et al., 2024b), Qwen2.5-VL-7B (Team, 2025), and the AIN-7B (Heakl et al., 2025). Traditional OCR approaches in our evaluation include Surya (Paruchuri, 2024b), Tesseract (Smith, 2007), EasyOCR (JaidedAI, 2020), and PaddleOCR (Li et al., 2022; Du et al., 2021). For specialized document processing tasks, we employ systems like Docling (Auer et al., 2024), and Marker (Paruchuri, 2024a). Layout detection capabilities are evaluated using methods implemented in Surya-layout (Paruchuri, 2024b), Yolo-doclaynet (Zhao et al., 2024) from MinerU (Wang et al., 2024a), and RT-DETR (Zhao et al., 2023) based method in Docling (Auer et al., 2024).

4.1 Evaluation Frameworks and Metrics

Our evaluation framework comprises nine specialized tasks designed to assess different aspects of Arabic OCR systems, as demonstrated in Figure 2. Each task addresses specific challenges in Arabic document processing. For this reason, we employ task-specific metrics to evaluate different aspects of document understanding.

PDF-to-Markdown: It evaluates the conversion of Arabic PDFs to structured markdown while preserving the text and table structure. Since both table and text structure are important, for evaluating PDF to Markdown conversion quality, we propose MARS (Markdown Recognition Score), which combines chrF (Popović, 2015) with Tree-Edit-Distance-based Similarity (TEDS) (Zhong et al., 2020):

\[
\text{MARS} = \alpha \cdot \text{chrF}_3 + (1-\alpha) \cdot \text{TEDS}(T_a, T_b)
\]

where \( \alpha \) (\( 0 \leq \alpha \leq 1 \)) is the weight. \( T_a \) represent

<table>
  <tr>
    <th>Dataset</th>
    <th>Metric</th>
    <th>Surya</th>
    <th>Yolo-doclaynet</th>
    <th>Detr (docling)</th>
  </tr>
  <tr>
    <td rowspan="4">BCE</td>
    <td>mAP@0.5</td>
    <td>0.506</td>
    <td>0.470</td>
    <td><b>0.750</b></td>
  </tr>
  <tr>
    <td>mAP@0.5:0.95</td>
    <td>0.381</td>
    <td>0.369</td>
    <td><b>0.566</b></td>
  </tr>
  <tr>
    <td>Precision</td>
    <td><b>0.751</b></td>
    <td>0.608</td>
    <td>0.626</td>
  </tr>
  <tr>
    <td>Recall</td>
    <td>0.593</td>
    <td>0.592</td>
    <td><b>0.725</b></td>
  </tr>
  <tr>
    <td>F1 Score</td>
    <td>0.635</td>
    <td>0.585</td>
    <td><b>0.654</b></td>
  </tr>
  <tr>
    <td rowspan="4">DocLayNet</td>
    <td>mAP@0.5</td>
    <td>0.675</td>
    <td>0.404</td>
    <td><b>0.758</b></td>
  </tr>
  <tr>
    <td>mAP@0.5:0.95</td>
    <td>0.469</td>
    <td>0.335</td>
    <td><b>0.541</b></td>
  </tr>
  <tr>
    <td>Precision</td>
    <td><b>0.782</b></td>
    <td>0.527</td>
    <td>0.635</td>
  </tr>
  <tr>
    <td>Recall</td>
    <td>0.856</td>
    <td>0.503</td>
    <td><b>0.770</b></td>
  </tr>
  <tr>
    <td>F1 Score</td>
    <td>0.799</td>
    <td>0.499</td>
    <td><b>0.670</b></td>
  </tr>
</table>

Table 4: Performance comparison of layout detection models using different evaluation metrics
predicted table structure and \( T_b \) the ground truth structure.

**Table Recognition:** We evaluate table extraction using both HTML and CSV formats, where HTML format (evaluated using TEDS (Zhong et al., 2020)) preserves rich structural information including cell spans and hierarchical relationships crucial for complex Arabic tables, while CSV format (evaluated using Jaccard Index 2) focuses on raw data extraction optimized for machine processing and data analysis pipelines. This dual-format evaluation ensures systems can both maintain complex table structures for human readability and provide clean, structured data for automated processing, specifically important for RAG based systems.

\[
J(P, G) = \frac{|P \cap G|}{|P \cup G|} = \frac{|P \cap G|}{|P| + |G| - |P \cap G|}
\]

where \( |P \cap G| \) represents the number of exact matching cells between predicted and ground truth tables, and \( |P \cup G| \) represents the total number of unique cells across both tables.

**Chart-to-Dataframe:** This task evaluates extracting structured data from Arabic charts into machine-readable dataframes. Systems must accurately parse numerical values, text labels, and preserve data relationships across chart types (bar, line, pie). We use the Structuring Chart-oriented Representation Metric (SCRM) (Xia et al., 2024)—which combines type recognition, topic understanding, and structural numerical fidelity (see Appendix D.1)—and also propose our own CharTeX (Chart Extraction Score) metric. CharTeX combines the chrF scores for chart type and topic with the jaccord index for the dataframe, using fuzzy matching (80% threshold) when columns do not exactly align.

\[
\text{Metric} = \alpha J_{type} + \beta J_{topic} + (1 - \alpha - \beta) J_{data}
\]

Here, \( J_{type} \) and \( J_{topic} \) denote the chrF scores between the predicted and ground-truth chart type and topic, while \( J_{data} \) measures the structural similarity of the predicted and ground-truth JSON data.

**Diagram-to-JSON:** This task evaluates the conversion of Arabic flowcharts and technical diagrams into JSON while preserving semantic relationships and technical specifications. We propose CODM (Code-Oriented Diagram Metric), extending SCRM (Xia et al., 2024), with the same formulation as in Eq 3. More detail about this metric is provided in Appendix D.1.

**Image-to-Text:** This task assess the basic text recognition capabilities across different Arabic fonts and styles, including the handling of cursive script connections, diacritical marks, and various text orientations. We use we use Character Error Rate (CER) and Word Error Rate (WER). For a predicted text sequence \( \hat{y} \) and ground truth sequence \( y \), CER is computed as: \( \text{CER} = \frac{\text{L}(y, \hat{y})}{|y|} \), where \( \text{L}(y, \hat{y}) \) is the Levenshtein distance between character sequences and \( |y| \) is the ground truth length. WER is calculated the same way with words as the unit of error.

**Visual Question Answering:** Tests the ability of models to understand and reason about Arabic document content, we evaluate using standard accuracy for MCQ questions and exact word match.

**Line Detection:** Focuses on the accurate identification and processing of individual text lines in Arabic documents. We evaluate using mean Average Precision (mAP) at different Intersection over Union (IoU) thresholds: mAP@0.5 and mAP@0.5:0.95, which assess the localization accuracy of detected text lines.

**Layout Detection:** Assesses document structure analysis capabilities, including the identification of headers, paragraphs, and complex layout elements in Arabic documents. Performance is measured using mAP@0.5 and mAP@0.5:0.95 for localization accuracy, complemented by Precision, Recall, and F1 scores to evaluate the overall detection quality across different layout components.

All metrics are computed on our diverse benchmark dataset, which encompasses various document types and complexity levels in both Arabic and multilingual contexts. Table 10 provides a detailed mapping of tasks, metrics, and evaluated systems.

4.2 Experimental Setup

We implement our evaluation pipeline with careful consideration of hyperparameters for different metrics. All experiments use NVIDIA A100 GPUs. For VLMs, we use their official implementations or API endpoints. Traditional OCR systems are evaluated using pre-trained models provided by the frameworks. For PDF-to-Markdown evaluation metric MARS 1, we choose \( \alpha = 0.5 \) and \( \alpha = 0.5 \) and \( \beta = 0.2 \) for Diagram-to-JSON evaluation metric CODM. We average the results over multiple runs, with performance comparisons shown in different tables (Table 3, 4, 5, 6, and 7).
<table>
  <tr>
    <th rowspan="2">Model Group</th>
    <th rowspan="2">Models</th>
    <th colspan="3">Table Extraction</th>
    <th colspan="3">End-to-End PDF</th>
  </tr>
  <tr>
    <th>TEDS (HTML)</th>
    <th>Jaccard (CSV)</th>
    <th>CHrF (Text)</th>
    <th>TEDS (Table)</th>
    <th>MARS</th>
  </tr>
  <tr>
    <td rowspan="3">Closed</td>
    <td>GPT-4o</td>
    <td><b>85.76</b></td>
    <td><b>66.36</b></td>
    <td>69.62</td>
    <td><b>60.61</b></td>
    <td>65.12</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>69.32</td>
    <td>49.50</td>
    <td>56.59</td>
    <td>52.69</td>
    <td>54.64</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>83.08</td>
    <td>65.55</td>
    <td><b>75.75</b></td>
    <td>55.55</td>
    <td><b>65.65</b></td>
  </tr>
  <tr>
    <td rowspan="3">Open</td>
    <td>Qwen2-VL-7B</td>
    <td>57.83</td>
    <td>40.20</td>
    <td>40.30</td>
    <td>2.54</td>
    <td>21.42</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>59.31</td>
    <td>59.58</td>
    <td>69.21</td>
    <td>11.65</td>
    <td>40.43</td>
  </tr>
  <tr>
    <td>AIN-7B</td>
    <td>75.94</td>
    <td>64.83</td>
    <td>56.52</td>
    <td>49.32</td>
    <td>52.92</td>
  </tr>
  <tr>
    <td rowspan="4">Framework</td>
    <td>Tesseract</td>
    <td>28.23<sup>D</sup></td>
    <td>14.85<sup>D</sup></td>
    <td>59.91<sup>D</sup></td>
    <td>45.44<sup>D</sup></td>
    <td>52.68<sup>D</sup></td>
  </tr>
  <tr>
    <td>EasyOCR</td>
    <td>49.10<sup>D</sup></td>
    <td>23.83<sup>D</sup></td>
    <td>57.46<sup>D</sup></td>
    <td>51.12<sup>D</sup></td>
    <td>54.29<sup>D</sup></td>
  </tr>
  <tr>
    <td>Surya</td>
    <td>39.09<sup>I</sup></td>
    <td>17.88<sup>I</sup></td>
    <td>58.38<sup>M</sup></td>
    <td>44.29<sup>M</sup></td>
    <td>51.34<sup>M</sup></td>
  </tr>
  <tr>
    <td>Docling (Auer et al., 2024) pipeline</td>
    <td>50.15<sup>M</sup></td>
    <td>70.42<sup>M</sup></td>
    <td>58.38<sup>M</sup></td>
    <td>44.29<sup>M</sup></td>
    <td>51.34<sup>M</sup></td>
  </tr>
  <tr>
    <td>Img2Table (Cattan, 2021) pipeline</td>
    <td>38.64<sup>I</sup></td>
    <td>16.04<sup>I</sup></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Paruchuri (Paruchuri, 2024a) pipeline</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Performance comparison of different models for table extraction and end-to-end PDF to markdown conversion tasks on our benchmark.

<table>
  <tr>
    <th rowspan="2">Group</th>
    <th rowspan="2">Models</th>
    <th colspan="3">OCR (image to text)</th>
  </tr>
  <tr>
    <th>CHrF ↑</th>
    <th>CER ↓</th>
    <th>WER ↓</th>
  </tr>
  <tr>
    <td rowspan="4">Closed</td>
    <td>GPT-4o</td>
    <td>61.01</td>
    <td>0.31</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>47.21</td>
    <td>0.43</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>77.95</td>
    <td><b>0.13</b></td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Azure</td>
    <td>50.97</td>
    <td>0.52</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td rowspan="6">Open</td>
    <td>Qwen2VL-7B</td>
    <td>33.94</td>
    <td>1.48</td>
    <td>1.55</td>
  </tr>
  <tr>
    <td>Qwen2.5VL-7B</td>
    <td>49.23</td>
    <td>1.20</td>
    <td>1.41</td>
  </tr>
  <tr>
    <td>AIN-7B</td>
    <td><b>78.33</b></td>
    <td>0.20</td>
    <td><b>0.28</b></td>
  </tr>
  <tr>
    <td>Qaari</td>
    <td>39.77</td>
    <td>1.80</td>
    <td>1.93</td>
  </tr>
  <tr>
    <td>Gemma3</td>
    <td>30.02</td>
    <td>1.05</td>
    <td>1.45</td>
  </tr>
  <tr>
    <td>ArabicNagout</td>
    <td>30.52</td>
    <td>4.37</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td rowspan="4">Framework</td>
    <td>Tesseract</td>
    <td>39.62</td>
    <td>0.54</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>EasyOCR</td>
    <td>45.47</td>
    <td>0.58</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Paddle</td>
    <td>16.73</td>
    <td>0.79</td>
    <td>1.02</td>
  </tr>
  <tr>
    <td>Surya</td>
    <td>20.61</td>
    <td>4.95</td>
    <td>5.61</td>
  </tr>
</table>

Table 6: Performance comparison of models for OCR (image to text) tasks on our benchmark. A detailed performance comparison among different open-source dataset is available in Appendix B

VQA-based accuracy metrics. Among closed-source models, Gemini-2.0 achieves the highest performance on chart understanding metrics, scoring 71.4% on SCRM and 56.28% on CharTeX. The performance gap between Gemini-2.0 and GPT-4o is particularly pronounced in CharTeX evaluation (10.33%) compared to SCRM (2.8%). Open-source models shows a significant limitation in complex chart understanding. While their SCRM scores remain competitive, both Qwen variants score below 23% on CharTeX evaluation. The visual question-answering results reveal an important exception to the general closed-source advantage. AIN achieves 87% on PATDVQA, surpassing Gemini-2.0 by 11.5%. AIN also shows competitive performance on MTVQA (31.50%), which is similar to GPT-4o and 4% better than GPT-4o-mini. This shows that open-source models can be competitive with closed-source alternatives.

5 Results and Discussion

In this section, we present a comprehensive evaluation of different models across different tasks of our framework. The results provide a clear distinction between the performance of closed-source models, open-source models, and framework-based solutions, revealing both their strengths and limitations. We observe very clear performance gap between closed and open-source solutions. While closed-source models like Gemini-2.0-Flash consistently outperform other models almost all the tasks.

5.1 Charts, Diagrams, and VQA

Table [7] presents model performance across different chart and diagram understanding tasks, evaluated using SCRM and CharTeX (for charts), and

5.2 Layout and Lines: Document Structure

Our evaluation of document structure understanding reveals distinct performance patterns across layout detection and line processing tasks. In layout detection (Table 4), RT-DETR (Zhao et al., 2023) achieves superior overall performance with mAP@0.5 scores of 0.750 and 0.758 on BCE (arabic only) and DocLayNet (english) dataset respectively. However, Surya (Paruchuri, 2024b) demonstrates higher precision (0.782 on DocLayNet, 0.751 on BCE), despite lower recall rates. This trade-off suggests that different architectures optimize for different aspects of layout detection. The line processing results (Table 3) highlight a clear contrast between detection and recognition
<table>
  <tr>
    <th rowspan="2">Group</th>
    <th rowspan="2">Model</th>
    <th colspan="2">Chart</th>
    <th colspan="2">Diagram</th>
    <th colspan="4">Visual QA</th>
  </tr>
  <tr>
    <th>SCRM</th>
    <th>CharTeX</th>
    <th>CODM</th>
    <th>MTVQA<sup>O</sup></th>
    <th>ChartsVQA<sup>M</sup></th>
    <th>DiagramsVQA<sup>M</sup></th>
    <th>PATDVQA<sup>M</sup></th>
    <th>Average</th>
  </tr>
  <tr>
    <td rowspan="3">Closed</td>
    <td>GPT-4o</td>
    <td>68.6</td>
    <td>45.95</td>
    <td>61.6</td>
    <td>32.00</td>
    <td>77.00</td>
    <td>85.29</td>
    <td>82.50</td>
    <td>69.19</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>67.2</td>
    <td>43.33</td>
    <td>61.4</td>
    <td>26.80</td>
    <td>58.00</td>
    <td>83.33</td>
    <td>80.00</td>
    <td>62.03</td>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash</td>
    <td>71.4</td>
    <td><b>56.28</b></td>
    <td><b>71.8</b></td>
    <td><b>35.00</b></td>
    <td>72.00</td>
    <td><b>88.24</b></td>
    <td>75.50</td>
    <td>67.68</td>
  </tr>
  <tr>
    <td rowspan="3">Open</td>
    <td>Qwen2-VL-7B</td>
    <td>56.6</td>
    <td>21.59</td>
    <td>63.0</td>
    <td>19.60</td>
    <td>59.00</td>
    <td>82.35</td>
    <td>77.50</td>
    <td>59.61</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B</td>
    <td>36.2</td>
    <td>22.08</td>
    <td>59.2</td>
    <td>23.00</td>
    <td>74.00</td>
    <td>79.41</td>
    <td>74.50</td>
    <td>62.72</td>
  </tr>
  <tr>
    <td>AIN-7B</td>
    <td>66.6</td>
    <td>34.61</td>
    <td>66.40</td>
    <td>31.50</td>
    <td>75.00</td>
    <td>85.29</td>
    <td><b>87.00</b></td>
    <td><b>69.69</b></td>
  </tr>
</table>

Table 7: Model Performance on Chart Understanding, Diagram Parsing, and Visual Question Answering Tasks. For VQA tasks, \( O \) denotes open-ended question type from MTVQA (Tang et al., 2024) dataset and \( M \) denotes MCQ type questions.

capabilities. While Surya excels in detection with a mAP@0.50 of 79.67%, EasyOCR demonstrates superior recognition performance (WER: 0.53, CER: 0.20). This inverse relationship between detection and recognition performance across models indicates a fundamental challenge in optimizing both capabilities simultaneously. Notably, Tesseract shows consistent but lower performance across both metrics, suggesting that newer architectures have made significant improvements over traditional approaches. We also observe that no single model excels at both detection and recognition, which requires for hybrid solutions.

5.3 Tables, OCR, and PDF-to-Markdown

Across table extraction tasks (Table 5), closed-source models maintain a clear advantage, with GPT-4o achieving 85.76% TEDS and 66.36% Jaccard scores. Among open-source models, AIN (75.94% TEDS) significantly outperforms Qwen variants, while specialized frameworks like Surya achieve competitive results (70.42% Jaccard) through targeted pipelines.
For OCR tasks, we evaluated GPT-4o (Hurst et al., 2024), Gemini-2.0-Flash (Google DeepMind, 2025), Azure OCR (Microsoft, 2024) in closed model; Qaari (NAMAA-Space, 2025), Gemma3 (Team et al., 2025), Arabic-Nagout (Rashad, 2024) and AIN (Heakl et al., 2025) in open source models and Tesseract (Smith, 2007), EasyOCR (JaidedAI, 2020), PaddleOCR (Li et al., 2022) and SuryaOCR (Paruchuri, 2024b) in frameworks (Table 6). Gemini-2.0-Flash leads with the lowest error rates (CER: 0.13, WER: 0.32). Notably, AIN matches this performance level (WER: 0.28), while traditional OCR frameworks like EasyOCR and Tesseract show moderate performance (CER: 0.58, 0.54). The significant performance drop in Paddle (CER: 0.79) and Surya (CER: 4.95) highlights the challenges in developing robust OCR systems.
End-to-end document processing (Table 5) reveals the largest gaps between approaches. Closed-source models maintain consistent performance (GPT-4o: 65.12% MARS, Gemini-2.0: 65.65% MARS), while open-source models show substantial degradation (Qwen2-VL-7B: 21.42% MARS). Framework approaches achieve better stability, with Tesseract and EasyOCR scoring above 50% MARS, suggesting that specialized pipelines can partially bridge the gap with larger models in complete document processing tasks.

![Bar chart showing ChrF by model on Arabic text variations](page_1012_1342_410_246.png)

Figure 5: ChrF by model on Arabic text variations

Our comprehensive evaluation demonstrates that while closed-source models maintain superior performance over open-source models across most tasks, specialized frameworks like Surya, RT-DETR Layout, and EasyOCR achieve competitive performance in targeted scenarios like table extraction, layout detection, and text recognition respectively. However, this framework advantage significantly diminishes in end-to-end pdf-to-markdown tasks where the integration capabilities of large models prove crucial, as evidenced by the performance gaps between commercial VLMs and tradit-
tional systems like EasyOCR, Surya and Tesseract in End-to-End PDF task (Table 5).

5.4 Performance on Challenging Cases
To evaluate model performance across different complexities of Arabic texts, we manually selected 104 samples representing four challenging categories: font variations, diacritics, text elongations, and tilted text. The ChrF score comparison (Figure 5) reveals distinct performance patterns across models, with GPT-4o demonstrating superior font handling (30.2) and leading in challenging tilted text recognition (13.1), while Azure OCR excels remarkably in diacritics recognition (40.5) and text elongations (35.2), indicating specialized Arabic script optimizations. The overall performance analysis shows GPT-4o leading at 26.0 average ChrF score, followed closely by Azure (26.3), Qwen2.5-VL-7B (25.3), and Gemini-2.0-Flash (24.7), while traditional OCR systems struggle significantly with Tesseract particularly challenged by diacritics (15.3) and tilted text (5.9). This analysis reveals that no single model excels across all Arabic text complexities, with specialized systems like Azure demonstrating domain-specific strengths in diacritics and elongation handling, while modern VLMs show more consistent performance but struggle with orientation variations, underscoring the need for Arabic-specific optimizations and highlighting the substantial performance gap between modern VLMs and traditional OCR approaches.

5.5 Model Performance across Chart Types

<table>
  <tr>
    <th>Chart Type</th>
    <th>Pie</th>
    <th>Dot</th>
    <th>Bar</th>
    <th>Histogram</th>
    <th>Line</th>
    <th>Dual Axis</th>
    <th>Density</th>
    <th>Area</th>
    <th>Choropleth</th>
    <th>Scatter</th>
    <th>Radial</th>
    <th>Grouped Bar</th>
    <th>Stacked Bar</th>
    <th>Heatmap</th>
    <th>Violin</th>
    <th>Box</th>
  </tr>
  <tr>
    <td>Gemini-2.0-Flash*</td>
    <td>72.4</td>
    <td>80.5</td>
    <td>86.9</td>
    <td>66.5</td>
    <td>78.3</td>
    <td>66.9</td>
    <td>66.7</td>
    <td>65.1</td>
    <td>68.7</td>
    <td>66.5</td>
    <td>65.1</td>
    <td>65.4</td>
    <td>65.3</td>
    <td>64.4</td>
    <td>21.5</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>64.5</td>
    <td>81.9</td>
    <td>82.7</td>
    <td>65.0</td>
    <td>81.7</td>
    <td>66.9</td>
    <td>65.4</td>
    <td>64.6</td>
    <td>68.1</td>
    <td>66.5</td>
    <td>65.1</td>
    <td>65.4</td>
    <td>65.3</td>
    <td>64.4</td>
    <td>22.5</td>
  </tr>
  <tr>
    <td>GPT-4o Mini*</td>
    <td>64.5</td>
    <td>81.9</td>
    <td>82.7</td>
    <td>65.0</td>
    <td>81.7</td>
    <td>66.9</td>
    <td>65.4</td>
    <td>64.6</td>
    <td>68.1</td>
    <td>66.5</td>
    <td>65.1</td>
    <td>65.4</td>
    <td>65.3</td>
    <td>64.4</td>
    <td>22.5</td>
  </tr>
  <tr>
    <td>Azure</td>
    <td>72.7</td>
    <td>80.5</td>
    <td>86.9</td>
    <td>66.5</td>
    <td>78.3</td>
    <td>66.9</td>
    <td>66.7</td>
    <td>65.1</td>
    <td>68.7</td>
    <td>66.5</td>
    <td>65.1</td>
    <td>65.4</td>
    <td>65.3</td>
    <td>64.4</td>
    <td>22.5</td>
  </tr>
  <tr>
    <td>Qwen2.5-VL-7B*</td>
    <td>72.7</td>
    <td>80.5</td>
    <td>86.9</td>
    <td>66.5</td>
    <td>78.3</td>
    <td>66.9</td>
    <td>66.7</td>
    <td>65.1</td>
    <td>68.7</td>
    <td>66.5</td>
    <td>65.1</td>
    <td>65.4</td>
    <td>65.3</td>
    <td>64.4</td>
    <td>22.5</td>
  </tr>
</table>

Figure 6: ChartEx results across different charts type.

The CharTeX evaluation across 16 different chart types reveals significant performance variations based on chart complexity and structural characteristics (Figure 6). Gemini-2.0-Flash demonstrates superior performance across most chart types, particularly excelling in simple geometric charts like Pie (72.4), Dot (72.7), and Bar Charts (69.2), while complex statistical visualizations like Violin Plots (32.3) and Box Plots (22.5) present significant challenges for all models. Simple chart types with clear boundaries consistently achieve higher scores across all models, with grouped and stacked bar charts showing intermediate performance levels around 40-50, indicating that while structural complexity affects extraction accuracy, the familiarity of bar chart formats provides some resilience. This pattern suggests that Arabic chart understanding faces particular difficulties with charts requiring statistical interpretation and continuous data representation, highlighting that current models perform best on charts with discrete, clearly separated data elements rather than continuous or overlapping visual representations.

6 Conclusion

We introduce a comprehensive benchmark for Arabic OCR that fills the gap in standardized evaluation frameworks for Arabic document processing. Our dataset of 8,809 samples across nine major domains is the most diverse collection assembled for OCR evaluation, incorporating handwritten, scanned, synthetic, and scene text, as well as complex tables, charts, and end-to-end pdf-to-markdown. This framework extends beyond simple text recognition to include structural document analysis and enables systematic assessment of OCR performance across various fonts, styles, and layouts.

7 Limitations and Future Directions

Despite its strengths, KITAB-Bench lacks coverage of low-resource dialects and institutional scans such as historical, governmental, and financial records. Future work should address OCR limitations in structural fidelity for tables and charts through richer datasets, refined metrics, and cross-lingual deep learning methods to enable robust and generalizable Arabic multimodal OCR. Moreover, current models often fail to generalize across domains and layouts, emphasizing the need for adaptable architectures and domain-specific fine-tuning.
References

Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, et al. 2023. Larabench: Benchmarking arabic ai with large language models. arXiv preprint arXiv:2305.14982.

Josh Achiham, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, and Muhammad Abdul-Mageed. 2024. Peacock: A family of arabic multimodal large language models and benchmarks. arXiv preprint arXiv:2403.01031.

Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas, Cesar Berrospil Ramis, Matteo Omenetti, Fabian Lindlbauer, Kasper Dinkla, et al. 2024. Docling technical report. arXiv preprint arXiv:2408.09869.

Gagan Bhatia, El Moatez Billah Nagoudi, Fakhraddin Alwajih, and Muhammad Abdul-Mageed. 2024. Qalam: A multimodal llm for arabic optical character and handwriting recognition. arXiv preprint arXiv:2407.13559.

Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhil Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018).

Houcine Boubaker, Abdelkarim Elbaati, Najiba Tagougui, Haikal El Abed, Monji Kherallah, Volker Märgner, and Adel M. Alimi. 2021. Adab database.

Hassina Bouressace and Janos Csirik. 2019. Printed arabic text database for automatic recognition systems. In Proceedings of the 2019 5th International Conference on Computer and Technology Applications, pages 107–111.

Xavier Cattan. 2021. img2table: Extract tables from images and scanned pdfs. https://github.com/xav ctn/img2table. Accessed: 2025-02-14.

H. Cheng, P. Zhang, S. Wu, et al. 2023. M6doc: A large-scale multi-format, multi-type, multi-layout, multi-language, multi-annotation category dataset for modern document layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muenninghoff, Kyle Lo, Luca Soldaini, et al. 2024. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146.

Yuning Du, Chenxia Li, Ruoyu Guo, Cheng Cui, Weiwei Liu, Jun Zhou, Bin Lu, Yehua Yang, Qiwen Liu, Xiaoguang Hu, et al. 2021. Pp-ocrv2: Bag of tricks for ultra lightweight ocr system. arXiv preprint arXiv:2109.03144.

Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. 2020. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.09941.

Husni A. El-Muhtaseb. 2010. Pats-a01 - an arabic text database. https://faculty.kfupm.edu.sa/ics/m uhtaseb/ArabicOCR/PATS-A01.htm. Database for Arabic Text Recognition Research.

Ali Elfilali. 2023. Hindawi books dataset. https://huggingface.co/datasets/Ali-C137/Hindawi-books-dataset. Dataset.

Ling Fu, Biao Yang, Zhebin Kuang, Jiajun Song, Yuzhe Li, Linghao Zhu, Qidi Luo, Xinyu Wang, Hao Lu, Mingxin Huang, et al. 2024. Ocrbench v2: An improved benchmark for evaluating large multimodal models on visual text localization and reasoning. arXiv preprint arXiv:2501.00321.

Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Sara Ghaboura, Ahmed Heakl, Omkar Thawakar, Ali Alharthi, Ines Riahi, Abduljalil Saif, Jorma Laaksonen, Fahad S Khan, Salman Khan, and Rao M Anwer. 2024. Camel-bench: A comprehensive arabic lmm benchmark. arXiv preprint arXiv:2410.18976.

Google DeepMind. 2025. Gemini Model Updates - February 2025. Accessed: 2025-02-14.

Heba Hassan, Ahmed El-Mahdy, and Mohamed E Hussein. 2021. Arabic scene text recognition in the deep learning era: Analysis on a novel dataset. IEEE Access, 9:107046–107058.

Ahmed Heakl, Sara Ghaboura, Omkar Thawakar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, and Salman Khan. 2025. Ain: The arabic inclusive large multimodal model. arXiv preprint arXiv:2502.00094.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

JaidedAI. 2020. Easyocr: Ready-to-use optical character recognition with multi-language support. https://github.com/JaidedAI/EasyOCR. Accessed: 2025-02-14.
Benjamin Kiessling, Daniel Stökl Ben Ezra, and Matthew Thomas Miller. 2019. Badam: A public dataset for baseline detection in arabic-script manuscripts. In Proceedings of the 5th International Workshop on Historical Document Imaging and Processing, page 13–18, New York, NY, USA. Association for Computing Machinery.

Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al. 2022. Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system. arXiv preprint arXiv:2206.03001.

Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, and Zhoujun Li. 2019. Tablebank: A benchmark dataset for table detection and recognition. arXiv preprint arXiv:1903.01949.

Minghao Li, Yiheng Xu, Leyang Cui, Shaohan Huang, Furu Wei, and Zhoujun Li. 2020. Docbank: A benchmark dataset for document layout analysis. arXiv preprint arXiv:2006.01038.

Yiren Li, Zheng Huang, Junchi Yan, Yi Zhou, Fan Ye, and Xianhui Liu. 2021. Gfte: graph-based financial table extraction. In Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part II, pages 644–658. Springer.

Nikolaos Livathinos, Cesar Berrospi, Maksym Lysak, Viktor Kuropiatnyk, Ahmed Nassar, Andre Carvalho, Michele Dolfi, Christoph Auer, Kasper Dinkla, and Peter Staar. 2021. Robust pdf document conversion using recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 15137–15145.

Sabri A Mahmoud, Irfan Ahmad, Wasfi G Al-Khatib, Mohammad Alshayeb, Mohammad Tanvir Parvez, Volker Märgner, and Gernot A Fink. 2014. Khatt: An open arabic offline handwritten text database. Pattern Recognition, 47(3):1096–1112.

Sabri A Mahmoud, Hamzah Luqman, Baligh M Al-Helali, Galal BinMakhshen, and Mohammad Tanvir Parvez. 2018. Online-khatt: an open-vocabulary database for arabic online-text processing. The Open Cybernetics & Systems Journal, 12(1).

Microsoft. 2024. OCR - Optical Character Recognition - Azure AI services. Accessed: 2025-05-27.

Hamdy Mubarak, Hend Al-Khalifa, and AbdulMohsen Al-Thubaity. 2022. Overview of osact5 shared task on arabic offensive language and hate speech detection. In Proceedings of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 162–166.

NAMAA-Space. 2025. Qari-ocr: A high-accuracy model for arabic optical character recognition. https://huggingface.co/collections/NAMAA-Space/qari-ocr-a-high-accuracy-model-for-arabic-optical-character-67c6cdf9584ef0684391335. Accessed: 2025-05-27.

Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. 2022. Tableformer: Table structure understanding with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4614–4623.

Shubham Singh Paliwal, D Vishwanath, Rohit Rahul, Monika Sharma, and Lovekesh Vig. 2019. Tablenet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 128–133. IEEE.

Werner Pantke, Martin Dennhardt, Daniel Fecker, Volker Märgner, and Tim Fingscheidt. 2014. An historical handwritten arabic dataset for segmentation-free word spotting-hadara80p. In 2014 14th International Conference on Frontiers in Handwriting Recognition, pages 15–20. IEEE.

Vik Paruchuri. 2024a. Marker: Convert pdf to markdown and other formats. https://github.com/VikParuchuri/marker.

Vik Paruchuri. 2024b. Surya: Accurate line-by-line text detection and recognition in complex documents. https://github.com/VikParuchuri/surya.

Mario Pechwitz, S Snoussi Maddouri, Volker Märgner, Noureddine Ellouze, Hamid Amiri, et al. 2002. Ifn/enit-database of handwritten arabic words. In Proc. of CIFED, volume 2, pages 127–136. Citeseer.

Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, and Peter W J Staar. 2022. Doclaynet: A large human-annotated dataset for document-layout analysis. arXiv preprint arXiv:2206.01062.

Maja Popović. 2015. chrft: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395.

Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, and Kavita Sultanpure. 2020. Cascadetabnet: An approach for end to end table detection and structure recognition from image-based documents. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 572–573.

Mohamed Rashad. 2024. Arabic-nougat: Fine-tuning vision transformers for arabic ocr and markdown extraction. arXiv preprint arXiv:2411.17835.

Rana SM Saad, Randa I Elanwar, NS Abdel Kader, Samia Mashali, and Margrit Betke. 2016. Bce-arabic-v1 dataset: Towards interpreting arabic document images for people with visual impairments. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pages 1–8.
M. Saeed, A. Chan, A. Mijar, and J. Moukarzel. 2024. Muhraf: Manuscripts of handwritten arabic dataset for cursive text recognition. arXiv preprint arXiv:2406.09630.

Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. 2017. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1162–1167. IEEE.

Zejiang Shen, Ruochen Zhang, Melissa Dell, Benjamin Charles Germain Lee, Jacob Carlson, and Weiming Li. 2021. Layoutparser: A unified toolkit for deep learning based document image analysis. In Document Analysis and Recognition–ICDAR 2021: 16th International Conference, Lausanne, Switzerland, September 5–10, 2021, Proceedings, Part I 16, pages 131–146. Springer.

Fouad Slimane, Rolf Ingold, Slim Kanoun, Adel M Alimi, and Jean Hennebert. 2009. A new arabic printed text image database and evaluation protocols. In 2009 10th international conference on document analysis and recognition, pages 946–950. IEEE.

Ray Smith. 2007. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE.

Peter WJ Staar, Michele Dolfi, Christoph Auer, and Costas Bekas. 2018. Corpus conversion service: A machine learning platform to ingest documents at scale. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 774–782.

Jingqun Tang, Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, Yanjie Wang, Yuliang Liu, Hao Liu, Xiang Bai, and Can Huang. 2024. Mtvqa: Benchmarking multilingual text-centric visual question answering. Preprint, arXiv:2405.11985.

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786.

Qwen Team. 2025. Qwen2.5-vl.

Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. 2024a. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Maurice Weber, Carlo Siebenschuh, Rory Butler, Anton Alexandrov, Valdemar Thanner, Georgios Tsolakis, Haris Jabbar, Ian Foster, Bo Li, Rick Stevens, et al. 2023. Wordscape: a pipeline to extract multilingual, visually rich documents with layout annotations from web crawl data. Advances in Neural Information Processing Systems, 36:26048–26068.

Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. 2024. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704.

Yue Wu and Prem Natarajan. 2017. Self-organized text detection with minimal post-processing via border learning. In International Conference on Computer Vision.

Renqiu Xia, Bo Zhang, Haoyang Peng, Hancheng Ye, Xiangchao Yan, Peng Ye, Botian Shi, Yu Qiao, and Junchi Yan. 2023. Structchart: Perception, structuring, reasoning for visual chart understanding. arXiv preprint arXiv:2309.11268.

Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvln: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185.

Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147.

Y Zhao, W Lv, S Xu, J Wei, G Wang, Q Dong, Y Liu, and J Chen. 2023. Detrs beat yolos on realtime object detection. arxiv e-prints. arXiv preprint arXiv:2304.08069.

Z. Zhao, H. Kang, B. Wang, and C. He. 2024. Doclayout-yolo: Enhancing document layout analysis through diverse synthetic data and global-to-local adaptive perception. arXiv preprint arXiv:2410.12628.

Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. 2021. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 697–706.

X Zhong, E ShafieiBavani, and A Jimeno-Yepes. 2019a. Image-based table recognition: data, model, and evaluation. corr abs/1911.10683. arXiv preprint arXiv:1911.10683.

Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. In European conference on computer vision, pages 564–580. Springer.

Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. 2019b. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), pages 1015–1022. IEEE.
A  Source of the Existing Dataset Collection

Our benchmark integrates diverse data sources to ensure comprehensive coverage of Arabic document types. As detailed in Table 2, the dataset combines manually curated samples, synthetic data generated through our LLM-assisted pipeline (Figure 4), and existing publicly available datasets. Key sources include:

• Handwritten Text: KHATT (paragraph and line-level annotations), ADAB, Muharaf, and OnlineKhatt.

• Historical Documents: HistoryAr and HistoricalBooks.

• Scene Text: EvAREST for real-world context diversity.

• Layout Analysis: BCE-Arabic-v1 and DocLayNet.

• Synthetic Content: 576 chart samples (16 types) and 422 diagram samples generated via our five-phase pipeline (Section 3.2).

The dataset emphasizes domain diversity, covering academic, medical, legal, financial, and technical documents. All samples underwent rigorous validation by native Arabic speakers to ensure linguistic and structural accuracy.

B  Detailed Performance Comparison

Table 9 provides granular performance metrics for VLMs and OCR frameworks across 12 Arabic text recognition datasets. Gemini-2.0-Flash demonstrates exceptional robustness on synthetic datasets (CER: 0.01 on PATS), while AIN-7B excels in historical manuscript recognition (CER: 0.26 on HistoryAr). Traditional OCR systems like Tesseract show limitations in handwritten text (CER: 1.26 on HistoryAr), highlighting the need for script-specific optimizations.

C  Data Analysis

Our data generation pipeline (Figure 4) produced 1,502 high-quality synthetic samples - comprising 576 graphs, 422 diagrams, and 456 tables, through LLM-assisted generation guided by domain-specific instructions (Figures 7 and 8) that ensured alignment with Arabic linguistic norms. During the human validation phase, 18% of initial outputs were discarded due to issues like right-to-left formatting errors and semantic inconsistencies. The resulting dataset offers diverse and balanced coverage, featuring 21 Arabic calligraphic styles, 36 sub-domains spanning financial reports to technical manuals, and complex structures such as merged cells in 43% of tables and dual-axis configurations in 29% of charts.

D  Evaluation Metrics

D.1  Tasks Models and Metrics

Table 10 maps evaluation tasks to corresponding models and metrics. The framework evaluates nine core capabilities:

• Structural Understanding: Layout detection (mAP), line detection (IoU)

• Content Extraction: Text recognition (CER), table parsing (TEDS)

• Semantic Reasoning: VQA accuracy, chart-to-dataframe conversion (SCRM)

• Specialized metrics like MARS (\( \alpha=0.5 \)) address the dual requirements of text fidelity and structural preservation in PDF-to-Markdown conversion.

D.2  Structuring Chart-oriented Representation Metric (SCRM)

The Structuring Chart-oriented Representation Metric (SCRM) evaluates chart understanding through three weighted components:

\[
\text{SCRM} = 0.4J_{type} + 0.3J_{topic} + 0.3J_{data}
\]

where \( J_{type} \) measures chart type recognition accuracy using Edit Distance, \( J_{topic} \) evaluates chart topic identification using Edit Distance, and \( J_{data} \) measures Mean Relative Error with and Error Thresholding criteria.

For entity comparison in \( J_{type} \) and \( J_{topic} \), we use the chrF character-based metric which captures partial matches effectively. For data comparison, value similarity is computed using relative error:

\[
e(p, q) = \frac{|Value_{pred}^p - Value_{GT}^q|}{Value_{GT}^q}
\]
D.3 Chart Extraction Score (CharTeX)

To evaluate chart data extraction quality, we propose CharTeX (Chart Extraction Score), which combines character-level text similarity with structural data assessment:

\[
\text{CharTeX} = \alpha J_{type} + \beta J_{topic} + (1 - \alpha - \beta)J_{data}
\] (5)

Where \( \alpha = 0.05 \) and \( \beta = 0.10 \) in our implementation, reflecting the relative importance of each component where \( J_{type} \) evaluates chart type recognition using chrF score (5% weight), \( J_{topic} \) assesses topic identification using chrF score (10% weight), and \( J_{data} \): measures structural data extraction accuracy using fuzzy matching (85% weight).
CharTeX improves upon SCRM by introducing structure-aware fuzzy matching (95% threshold) and leveraging the Hungarian algorithm for optimal alignment. In contrast to SCRM’s reliance on (*entity, value*) triplet matching, CharTeX incorporates column-level semantics and chrF-based scoring, enhancing robustness to text variations and structural discrepancies, particularly critical for Arabic charts with complex layouts. This design mitigates SCRM’s sensitivity to superficial mismatches and its disregard for tabular structure.

D.4 Markdown Recognition Score (MARS)

To evaluate the quality of PDF-to-Markdown conversion, we propose the Markdown Recognition Score (MARS), defined as:

\[
\text{MARS} = \alpha \cdot \text{chrF3} + (1 - \alpha) \cdot \text{TEDS}(T_a, T_b)
\]

where \( \alpha \in [0, 1] \) is set to 0.5 to balance text fidelity and structural accuracy. Here, \( T_a \) and \( T_b \) denote the predicted and ground truth table structures, respectively.
MARS jointly captures character-level accuracy using chrF3, ideal for OCR tasks requiring fine-grained text recognition, and hierarchical layout preservation via TEDS, which quantifies the tree-edit distance between table structures. By assigning equal weight to both components, MARS offers a robust metric that reflects both semantic and structural fidelity in document conversion. As both chrF3 and TEDS are established in prior work, MARS inherits their theoretical validity without the need for further empirical justification.

D.5 Code-Oriented Diagram Metric (CODM)

The Code-Oriented Diagram Metric (CODM) extends SCRM with a graph-theoretic foundation specifically designed for diagrams where structural relationships are paramount:

\[
\text{CODM} = 0.5J_{topology} + 0.2J_{topic} + 0.3J_{semantics}
\] (6)

Where \( J_{topology} \) evaluates diagram type (50%) using edit distance, \( J_{topic} \) assesses topic identification (20%) using edit distance, and \( J_{semantics} \) measures diagram structure through Graph Edit Distance (30%).

This metric converts both predicted and ground truth diagram data into graph structures, where nodes represent entities and edges represent relationships. This approach effectively evaluates both node-edge relationships and semantic labels in technical diagrams such as flowcharts, class diagrams, and sequence diagrams.
Further, domain-specific prompts are used to guide model responses for accurate metric calculation. For instance, sequence diagrams require strict adherence to Arabic UML notation standards during evaluation, ensuring fair assessment across different diagram conventions.
<table>
  <tr>
    <th>Domain</th>
    <th>Sub-Domain</th>
    <th>Dataset Source</th>
    <th>Original</th>
    <th>Selected</th>
    <th>Total</th>
  </tr>
  <tr>
    <td rowspan="2">PDF to Markdown</td>
    <td>General</td>
    <td>Manual</td>
    <td>33</td>
    <td>33</td>
    <td>33</td>
  </tr>
  <tr>
    <td colspan="5"></td>
  </tr>
  <tr>
    <td rowspan="2">Layout Detection</td>
    <td>Docs</td>
    <td>BCE-Arabic-v1 (Saad et al., 2016)</td>
    <td>1.9k</td>
    <td>1,700</td>
    <td>2,100</td>
  </tr>
  <tr>
    <td></td>
    <td>DocLayNet (Plitzmann et al., 2022)</td>
    <td>80k</td>
    <td>400</td>
    <td></td>
  </tr>
  <tr>
    <td>Line Detection</td>
    <td>Docs</td>
    <td>Manual</td>
    <td>375</td>
    <td>378</td>
    <td>378</td>
  </tr>
  <tr>
    <td>Line Recognition</td>
    <td>Docs</td>
    <td>Manual</td>
    <td>375</td>
    <td>378</td>
    <td>378</td>
  </tr>
  <tr>
    <td>Table Recognition</td>
    <td>Financial</td>
    <td>Pixmo (Deitke et al., 2024)</td>
    <td>490</td>
    <td>456</td>
    <td>456</td>
  </tr>
  <tr>
    <td rowspan="11">Image to Text</td>
    <td>Synthetic</td>
    <td>PATS (El-Muhtaseb, 2010)</td>
    <td>21.6k</td>
    <td>500</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>SythenAR</td>
    <td>39.1k</td>
    <td>500</td>
    <td></td>
  </tr>
  <tr>
    <td>Historical</td>
    <td>HistoryAr (Pantke et al., 2014)</td>
    <td>1.5k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>HistoricalBooks</td>
    <td>40</td>
    <td>10</td>
    <td></td>
  </tr>
  <tr>
    <td>Hand. Paragraph</td>
    <td>Khatt (Mahmoud et al., 2014)</td>
    <td>2.72k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td>Hand. Word</td>
    <td>ADAB (Boubaker et al., 2021)</td>
    <td>15k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Muharaf (Saeed et al., 2024)</td>
    <td>24.5k</td>
    <td>200</td>
    <td>3,760</td>
  </tr>
  <tr>
    <td>Hand. Line</td>
    <td>OnlineKhatt (Mahmoud et al., 2018)</td>
    <td>8.5k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Khatt (Mahmoud et al., 2014)</td>
    <td>13.4k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td>PPT</td>
    <td>ISI-PPT (Wu and Natarajan, 2017)</td>
    <td>86.5k</td>
    <td>500</td>
    <td></td>
  </tr>
  <tr>
    <td>Blogs</td>
    <td>ArabicOCR</td>
    <td>20.3k</td>
    <td>50</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Hindawi (Elfilali, 2023)</td>
    <td>79k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>EyAREST (Hassan et al., 2021)</td>
    <td>5.59k</td>
    <td>800</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="12">Charts to DataFrame</td>
    <td>Bar</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>61</td>
    <td></td>
  </tr>
  <tr>
    <td>Line</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>43</td>
    <td></td>
  </tr>
  <tr>
    <td>Pie</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>56</td>
    <td></td>
  </tr>
  <tr>
    <td>Box</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>31</td>
    <td></td>
  </tr>
  <tr>
    <td>Violin</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>36</td>
    <td></td>
  </tr>
  <tr>
    <td>Area</td>
    <td>Synthetic</td>
    <td>50</td>
    <td>29</td>
    <td></td>
  </tr>
  <tr>
    <td>SunBurst</td>
    <td>Synthetic</td>
    <td>30</td>
    <td>15</td>
    <td></td>
  </tr>
  <tr>
    <td>Dot</td>
    <td>Synthetic</td>
    <td>30</td>
    <td>15</td>
    <td></td>
  </tr>
  <tr>
    <td>Dual Axis</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>26</td>
    <td>576</td>
  </tr>
  <tr>
    <td>Density Curve</td>
    <td>Synthetic</td>
    <td>10</td>
    <td>5</td>
    <td></td>
  </tr>
  <tr>
    <td>Bubble</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>13</td>
    <td></td>
  </tr>
  <tr>
    <td>Grouped Bar</td>
    <td>Synthetic</td>
    <td>50</td>
    <td>60</td>
    <td></td>
  </tr>
  <tr>
    <td>Stacked Bar</td>
    <td>Synthetic</td>
    <td>50</td>
    <td>82</td>
    <td></td>
  </tr>
  <tr>
    <td>Histogram</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>70</td>
    <td></td>
  </tr>
  <tr>
    <td>HeatMap</td>
    <td>Synthetic</td>
    <td>10</td>
    <td>11</td>
    <td></td>
  </tr>
  <tr>
    <td>Scatter</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>23</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="6">Diagram to Json</td>
    <td>Sequence</td>
    <td>Synthetic</td>
    <td>50</td>
    <td>46</td>
    <td></td>
  </tr>
  <tr>
    <td>Funnel</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>52</td>
    <td></td>
  </tr>
  <tr>
    <td>Class</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>30</td>
    <td></td>
  </tr>
  <tr>
    <td>Network</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>18</td>
    <td>226</td>
  </tr>
  <tr>
    <td>Venn</td>
    <td>Synthetic</td>
    <td>20</td>
    <td>7</td>
    <td></td>
  </tr>
  <tr>
    <td>FlowChart</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>112</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>TreeMap</td>
    <td>Synthetic</td>
    <td>100</td>
    <td>157</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">VQA</td>
    <td>Diagrams</td>
    <td>Manual</td>
    <td>102</td>
    <td>102</td>
    <td></td>
  </tr>
  <tr>
    <td>Charts</td>
    <td>Manual</td>
    <td>105</td>
    <td>100</td>
    <td>902</td>
  </tr>
  <tr>
    <td>News Letter</td>
    <td>PATD (Bouressace and Csirik, 2019)</td>
    <td>2.42k</td>
    <td>200</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Scene</td>
    <td>MTVQA</td>
    <td>818</td>
    <td>500</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4">Total Dataset Size</td>
    <td>--</td>
    <td>8,809</td>
  </tr>
</table>

Table 8: Dataset Distribution Across Different Domains, sub-domains and Data Source
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Size</th>
    <th colspan="4">GPT-4o</th>
    <th colspan="4">GPT-4o-mini</th>
    <th colspan="4">Azure OCR</th>
    <th colspan="4">Gemini-2.0-Flash</th>
    <th colspan="2">Qwen2-VL</th>
  </tr>
  <tr>
    <th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th>
  </tr>
  <tr>
    <td>PATS</td><td>500</td><td>0.23</td><td>0.30</td><td>0.53</td><td>0.71</td><td>0.03</td><td>0.10</td><td>0.01</td><td>0.02</td><td>1.02</td><td>1.02</td>
  </tr>
  <tr>
    <td>SythenAR</td><td>500</td><td>0.09</td><td>0.20</td><td>0.14</td><td>0.32</td><td>0.10</td><td>0.27</td><td>0.07</td><td>0.17</td><td>0.59</td><td>1.13</td>
  </tr>
  <tr>
    <td>HistoryAr</td><td>200</td><td>0.51</td><td>0.82</td><td>0.67</td><td>0.96</td><td>0.24</td><td>0.68</td><td>0.28</td><td>0.64</td><td>3.46</td><td>2.86</td>
  </tr>
  <tr>
    <td>HistoricalBooks</td><td>10</td><td>0.41</td><td>0.76</td><td>0.59</td><td>0.88</td><td>0.29</td><td>0.71</td><td>0.05</td><td>0.22</td><td>1.90</td><td>2.16</td>
  </tr>
  <tr>
    <td>Khatt</td><td>200</td><td>0.45</td><td>0.74</td><td>0.64</td><td>0.91</td><td>0.83</td><td>0.92</td><td>0.19</td><td>0.45</td><td>1.12</td><td>5.04</td>
  </tr>
  <tr>
    <td>Adab</td><td>200</td><td>0.30</td><td>0.73</td><td>0.35</td><td>0.83</td><td>0.99</td><td>0.99</td><td>0.19</td><td>0.56</td><td>0.63</td><td>1.08</td>
  </tr>
  <tr>
    <td>Muharaf</td><td>200</td><td>0.56</td><td>0.90</td><td>0.63</td><td>0.94</td><td>0.52</td><td>0.82</td><td>0.33</td><td>0.69</td><td>3.57</td><td>2.87</td>
  </tr>
  <tr>
    <td>OnlineKhatt</td><td>200</td><td>0.29</td><td>0.63</td><td>0.41</td><td>0.76</td><td>0.72</td><td>0.85</td><td>0.17</td><td>0.44</td><td>1.30</td><td>2.01</td>
  </tr>
  <tr>
    <td>ISI-PPT</td><td>500</td><td>0.08</td><td>0.18</td><td>0.15</td><td>0.31</td><td>0.98</td><td>0.98</td><td>0.06</td><td>0.15</td><td>1.03</td><td>1.06</td>
  </tr>
  <tr>
    <td>ArabicOCR</td><td>50</td><td>0.06</td><td>0.26</td><td>0.16</td><td>0.46</td><td>0.01</td><td>0.11</td><td>0.00</td><td>0.02</td><td>1.25</td><td>1.50</td>
  </tr>
  <tr>
    <td>Hindawi</td><td>200</td><td>0.34</td><td>0.56</td><td>0.48</td><td>0.71</td><td>0.06</td><td>0.28</td><td>0.01</td><td>0.04</td><td>1.82</td><td>2.05</td>
  </tr>
  <tr>
    <td>EvArest</td><td>800</td><td>0.20</td><td>0.38</td><td>0.25</td><td>0.51</td><td>0.32</td><td>0.50</td><td>0.18</td><td>0.36</td><td>0.41</td><td>0.95</td>
  </tr>
  <tr>
    <td></td><td>3,760</td><td>0.31</td><td>0.55</td><td>0.43</td><td>0.71</td><td>0.52</td><td>0.69</td><td>0.13</td><td>0.32</td><td>1.48</td><td>1.20</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Size</th>
    <th colspan="4">Qwen2.5-VL</th>
    <th colspan="4">AIN</th>
    <th colspan="4">Qari</th>
    <th colspan="4">Tesseract</th>
    <th colspan="4">Surya</th>
    <th colspan="2">Paddle</th>
  </tr>
  <tr>
    <th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th><th>CER</th><th>WER</th>
  </tr>
  <tr>
    <td>PATS</td><td>500</td><td>0.98</td><td>1.03</td><td>0.26</td><td>0.36</td><td>0.00</td><td>0.00</td><td>0.14</td><td>0.28</td><td>4.66</td><td>4.67</td><td>0.77</td><td>1.00</td>
  </tr>
  <tr>
    <td>SythenAR</td><td>500</td><td>1.68</td><td>1.69</td><td>0.21</td><td>0.40</td><td>0.04</td><td>0.16</td><td>0.31</td><td>0.72</td><td>4.82</td><td>7.90</td><td>0.80</td><td>1.01</td>
  </tr>
  <tr>
    <td>HistoryAr</td><td>200</td><td>3.48</td><td>3.39</td><td>0.47</td><td>0.83</td><td>0.26</td><td>0.54</td><td>0.72</td><td>1.26</td><td>10.32</td><td>12.78</td><td>0.79</td><td>1.01</td>
  </tr>
  <tr>
    <td>HistoricalBooks</td><td>10</td><td>0.67</td><td>0.97</td><td>0.33</td><td>0.72</td><td>0.84</td><td>0.88</td><td>0.74</td><td>0.99</td><td>6.81</td><td>6.30</td><td>0.71</td><td>1.00</td>
  </tr>
  <tr>
    <td>Khatt</td><td>200</td><td>1.60</td><td>1.80</td><td>0.07</td><td>0.22</td><td>0.61</td><td>1.12</td><td>0.67</td><td>1.06</td><td>4.25</td><td>3.77</td><td>0.76</td><td>1.00</td>
  </tr>
  <tr>
    <td>Adab</td><td>200</td><td>0.91</td><td>1.11</td><td>0.00</td><td>0.01</td><td>1.00</td><td>1.00</td><td>1.00</td><td>1.14</td><td>7.28</td><td>8.71</td><td>0.88</td><td>1.15</td>
  </tr>
  <tr>
    <td>Muharaf</td><td>200</td><td>2.40</td><td>2.74</td><td>0.61</td><td>0.96</td><td>0.38</td><td>0.54</td><td>0.77</td><td>1.22</td><td>6.19</td><td>7.48</td><td>0.80</td><td>1.01</td>
  </tr>
  <tr>
    <td>OnlineKhatt</td><td>200</td><td>1.52</td><td>1.53</td><td>0.36</td><td>0.70</td><td>0.03</td><td>0.12</td><td>0.59</td><td>1.20</td><td>6.71</td><td>6.95</td><td>0.78</td><td>1.03</td>
  </tr>
  <tr>
    <td>ISI-PPT</td><td>500</td><td>1.27</td><td>1.39</td><td>0.36</td><td>0.54</td><td>0.52</td><td>0.53</td><td>0.31</td><td>0.64</td><td>4.25</td><td>3.77</td><td>0.81</td><td>1.03</td>
  </tr>
  <tr>
    <td>ArabicOCR</td><td>50</td><td>0.02</td><td>0.08</td><td>1.00</td><td>1.00</td><td>0.01</td><td>0.01</td><td>0.01</td><td>0.01</td><td>2.75</td><td>3.58</td><td>0.77</td><td>1.00</td>
  </tr>
  <tr>
    <td>Hindawi</td><td>200</td><td>0.27</td><td>0.42</td><td>1.00</td><td>1.00</td><td>0.11</td><td>0.15</td><td>0.31</td><td>0.72</td><td>0.15</td><td>0.20</td><td>0.76</td><td>1.00</td>
  </tr>
  <tr>
    <td>EvArest</td><td>800</td><td>4.65</td><td>4.75</td><td>0.19</td><td>0.36</td><td>0.30</td><td>0.32</td><td>0.85</td><td>1.02</td><td>5.91</td><td>3.86</td><td>0.89</td><td>1.04</td>
  </tr>
  <tr>
    <td></td><td>3,760</td><td>1.80</td><td>1.93</td><td>0.28</td><td>0.54</td><td>0.20</td><td>0.58</td><td>0.89</td><td>0.79</td><td>4.95</td><td>5.61</td><td>0.79</td><td>1.02</td>
  </tr>
</table>

Table 9: Performance comparison of Large Vision-Language Models on KITAB-Bench (lower is better).
<table>
  <tr>
    <th>Task</th>
    <th>Metrics</th>
    <th>Open LLMs</th>
    <th>Closed LLMs</th>
    <th>OCR Systems</th>
  </tr>
  <tr>
    <th colspan="5">Document Understanding Tasks</th>
  </tr>
  <tr>
    <td>PDF to Markdown</td>
    <td>chrF + TEDS</td>
    <td>–</td>
    <td>–</td>
    <td>Docling<br>Marker<br>MinerU<br>PDF-Extract-Kit</td>
  </tr>
  <tr>
    <td>Layout Detection</td>
    <td>mAP@0.5<br>mAP@0.5:0.95<br>Precision<br>Recall<br>F1</td>
    <td>–</td>
    <td>–</td>
    <td>Surya<br>Yolo-doclaynet (MinerU)<br>Detr (docling)</td>
  </tr>
  <tr>
    <td>Line Detection</td>
    <td>mAP@0.5<br>mAP@0.5:0.95</td>
    <td>–</td>
    <td>–</td>
    <td>Surya<br>Tesseract<br>EasyOCR</td>
  </tr>
  <tr>
    <td>Line Recognition</td>
    <td>WER, CER</td>
    <td>–</td>
    <td>–</td>
    <td>Surya<br>Tesseract<br>EasyOCR</td>
  </tr>
  <tr>
    <th colspan="5">Table Understanding Tasks</th>
  </tr>
  <tr>
    <td>Tables Recognition (HTML)</td>
    <td>TEDS (<a href="https://arxiv.org/abs/1908.07426">Zhong et al., 2019a</a>)</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>Docling[EasyOCR]<br>Docling[Tesseract]<br>Marker<br>Img2Table[EasyOCR]<br>Img2Table[Tesseract]</td>
  </tr>
  <tr>
    <td>Tables Recognition (CSV)</td>
    <td>Jaccard Index</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>Docling[EasyOCR]<br>Docling[Tesseract]<br>Marker<br>Img2Table[EasyOCR]<br>Img2Table[Tesseract]</td>
  </tr>
  <tr>
    <th colspan="5">Visual Understanding Tasks</th>
  </tr>
  <tr>
    <td>Image to Text</td>
    <td>CER, WER<br>chrF, BLEU<br>METEOR</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN-7B<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>Docling[EasyOCR]<br>Docling[Tesseract]<br>Marker<br>Img2Table[EasyOCR]<br>Img2Table[Tesseract]</td>
  </tr>
  <tr>
    <td>Charts to DataFrame</td>
    <td>SCRM (<a href="https://arxiv.org/abs/2310.14222">Xia et al., 2024, 2023</a>)</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>–</td>
  </tr>
  <tr>
    <td>Diagram to Json</td>
    <td>SCRM</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN-7B<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>–</td>
  </tr>
  <tr>
    <td>VQA</td>
    <td>Accuracy + Word Match Score</td>
    <td>Qwen2-VL<br>Qwen2.5-VL<br>AIN-7B<br>PaliGemma</td>
    <td>GPT-4o<br>GPT-4o-mini<br>Gemini-2.0-Flash</td>
    <td>–</td>
  </tr>
</table>

Table 10: Comprehensive evaluation metrics and models for document understanding tasks. The table is organized into three main categories: document understanding, table understanding, and visual understanding tasks. Each task is evaluated using specific metrics and implemented across various models and OCR systems.
Charts: Type Prompt
***You are an expert in detecting chart types. Below are examples of the expected output format:

Example 1:
bar chart

Example 2:
scatter chart

Example 3:
histogram

Your task is to determine the type of chart shown in the given image.

**Instructions:**
- **Respond with only the chart type** (e.g., 'bar chart', 'scatter chart')
- **Do not include any additional text, explanations, or descriptions.**
- **Ensure the output matches the format in the examples exactly.**

Provide only the chart type in **single quotes** as shown in the examples above.

What type of chart is shown in the image? Don’t output any extra text***

PDF to Markdown Prompt
***Extract the text from the document in Markdown format, and extract the tables in HTML format.
Do not add style or anything, just the text. Do not ever generate tables in markdown format. Give me the output, nothing else.***

OCR Prompt
***Extract the text in the image. Give me the final text, nothing else.***

Charts: Topic Prompt
مثال في تحفيز وتشجيع الخطط البيانية. فيما يلي أسلوب توجيه نصي الإجابة النموذجي
مثال 1
أثر أكبر مساحة حسب النوع الأول
مثال 2
أثر الصنادل حول الموضوعات التي تتحدث في الأطباء
مثال 3
عدد موضوع أو محتوى الخطط البيانية فقط
مثال 4
عدد الأراء العامة المدونة
مثال 5
عدد التنسيق المحدد دون إضافة أي شرح أو تفسير إضافي
ما هو موضوع أو محتوى الخطط البيانية؟

Charts: Data Prompt
***You are an expert in chart data extraction. You are given a chart image and you should provide the chart data in CSV format.
Here are some examples:

Example 1:
[CSV table]

Example 2:
[CSV table]

Not give me the results as in the previous CSV format***

Diagrams: Type Prompt
***You are an expert in detecting chart types. Below are examples of the expected output format:

Example 1:
treemap

Example 2:
flowchart

Example 3:
diagram

Your task is to determine the type of chart shown in the given image.

**Instructions:**
- **Respond with only the chart type** (e.g., 'flowchart', 'sequence').
- **Do not provide any explanations, descriptions, or additional text.**
- **Ensure the output strictly follows the format shown in the examples.**

What type of chart is shown in the image?***

Diagrams: Topic Prompt
***You are an expert in detecting chart types. Below are examples of the expected output format:

Example 1:
treemap

Example 2:
flowchart

Example 3:
diagram

Your task is to determine the type of chart shown in the given image.

**Instructions:**
- **Respond with only the chart type** (e.g., 'flowchart', 'sequence').
- **Do not provide any explanations, descriptions, or additional text.**
- **Ensure the output strictly follows the format shown in the examples.**

What type of chart is shown in the image?***

Figure 7: Prompts for Different Task Categories.
Diagrams: Data Prompt

'''You are an expert in diagram data extraction. Your task is to analyze the given diagram and generate structured data in JSON format that captures nodes (entities) and edges (relationships).
## Output Format Example:
for flowchart:
json
{
    "nodes": [
        {
            "id": "1",
            "text": "جمع البيانات",
            "description": "جميع البيانات المدفوعة من النظام المصدر"
        },
        {
            "id": "2",
            "text": "مرفق البيانات",
            "description": "مرتكز البيانات لمواد ملائمة إعداد التصور في فتحة 3"
        },
        {
            "id": "3",
            "text": "تحليل البيانات",
            "description": "تحليل البيانات في القائمة للصور في جداول الإشارات"
        }
    ],
    "edges": [
        {
            "from": "1",
            "to": "2",
            "text": ""
        },
        {
            "from": "2",
            "to": "3",
            "text": ""
        },
        {
            "from": "3",
            "to": "4",
            "text": "statistical analysis"
        }
    ]
}
treemap:
json
{
    [...]
}
class diagram:
json
{
    [...]
}
'''

Table: HTML Prompt

'''Extract the data from the table below and provide the output in HTML format. Output only the data as HTML and nothing else. Here is one example:
<html>
<table>
  <thead>
    <tr>
      <th>المصدر</th>
      <th>الabad</th>
      <th>اسم الوسيلة</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>شركة سلطان خدمات يحيى الطائي</td>
      <td>15-06-28,2023</td>
      <td>إمدادات انتاجية فائق الجودة</td>
    </tr>
    <tr>
      <td>شركة المنار للسياحة</td>
      <td>20-04-15,2023</td>
      <td>مونديا كويSuit</td>
    </tr>
    <tr>
      <td>مصرف الراجحي</td>
      <td>03-12,2023</td>
      <td> необходимо ضمان</td>
    </tr>
    <tr>
      <td>سيب بناء في الصخيرات</td>
      <td>01-09-35,2023</td>
      <td>كيفية تقييم الأداء</td>
    </tr>
  </tbody>
</table>
'''
Now generate the data for the provided table.''''

Table: Dataframe Prompt

'''Extract the data from the table below and provide the output in CSV format. Output only the data as CSV and nothing else. Here is one example:
csv
اسم الشركة والمحل с таблица (ملف roller يعيش)
أعمال المصدر على شركة يحيى اللطيف
الاستثمار: 15-06-28، 2023
Deprecated
لا يمكن تعيين foundational كعمليات
علاقت الأعمال بشركة كويSuit
يتطلب كويSuit أن تمتلك
حول كويSuit إلى شركة_BI نقل
يتم إجراؤها في شركاتها مع شريك مجموعة
البناء: 01-09-35، 2023
...
Now generate the data for the provided table.'''

Reviewer paper summaries: This paper introduces KITAB-Bench, a comprehensive multi-domain benchmark designed specifically for Arabic OCR and document understanding. The authors compile a dataset of 8,809 samples spanning nine major domains and 36 sub-domains. These include handwritten text, scanned documents, complex tables, charts, diagram parsing, and PDF-to-Markdown conversion. In addition to curating a diverse dataset (with both real and synthetic samples), the paper proposes several novel evaluation metrics—such as MARS for PDF-to-Markdown, CharTeX for chart extraction, and CODM for diagram analysis—to better capture the unique challenges posed by Arabic script (e.g., cursiveness, right-to-left flow, and complex typography). Extensive experiments compare closed-source vision-language models (like GPT-4 and Gemini-2.0-Flash) against open-source OCR systems and specialized frameworks, revealing significant performance gaps and highlighting areas for further improvement.

The paper proposes a new benchmark datasets for several arabic ocr tasks specifically Table Recognition, Chart to Dataframe, Image to Text, Diagram to JSON, VQA, Line Detection/Recognition, Layout Detection, PDF to Markdown. 

They provide a clear flow of how they generated their different datasets using LLMs and human in the loop. They introduce benchmarks on all 9 tasks using open sourced, closed sourced and frameworks. 

They introduce MARS (Markdown Recognition Score) for PDF to markdown evaluation.
Summary of strengths: - Comprehensive Dataset: The benchmark covers a wide range of document types and domains, which is particularly valuable given the scarcity of large-scale Arabic OCR resources.
- Novel Evaluation Metrics: The introduction of task-specific metrics (e.g., MARS, CharTeX, CODM) is innovative and tailored to address the intricacies of Arabic document structure.
- Extensive Empirical Analysis: Detailed experimental evaluations across multiple tasks and systems (both open- and closed-source) provide clear insights into current performance limitations and set the stage for future research.
- By focusing on a language with unique challenges, the work fills a significant gap in the OCR and document understanding literature and will likely benefit a broad community working on Arabic language processing.

The presentation is good and the flow of the paper is easy to follow.

They present a benchmark on a diverse yet large set of tasks in Arabic OCR.

They provide a clear illustration of how the dataset was generated and validated.
Summary of weaknesses: - Despite the ambitious scope, the paper fails to delve deeply into any one aspect of Arabic OCR. The analysis is perfunctory at best, with many claims unsupported by adequate empirical evidence. Critical elements, such as handling historical manuscripts and low-resource dialects, are glaringly neglected.
- A lack of focus mars the promise of a comprehensive multi-domain benchmark. The paper tries to cover too much without the necessary depth or quality in any one area, leading to a work that is both overly ambitious and critically underdeveloped.
- The introduction of metrics like MARS, CharTeX, and CODM is neither rigorously motivated nor theoretically substantiated. The paper provides little to no comparative analysis against established metrics, making it unclear whether these measures truly offer any improvement over existing standards.

The paper lacks comparisons with recent fine-tuned arabic ocr models like DeepOnKHATT[1]

The authors don't show how their datasets differ from previous Arabic benchmarks for every specific task.

The paper also lacks any error analysis. Why did the models fail to identify the sample? (is it a recurring pattern or is the model just hallucinating?).  I think this would strengthen the paper’s findings
 
1 Alwajih, Fakhraddin, et al. "DeepOnKHATT: an end-to-end Arabic online handwriting recognition system." International Journal of Pattern Recognition and Artificial Intelligence 35.11 (2021): 2153006.
Comments, suggestions and typos: A publicly available repository with code, data splits, and detailed documentation would greatly enhance the reproducibility and impact of the work.

n/a
Overall assessment scores: 2.5

3.5