KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and
  Document Understanding
Authors: name: Ahmed Heakl; name: Abdullah Sohail; name: Mukul Ranjan; name: Rania Hossam; name: Ghazi Ahmed; name: Mohamed El-Geish; name: Omar Maher; name: Zhiqiang Shen; name: Fahad Khan; name: Salman Khan
Published: 2025-02-20 18:41:23

Abstract: With the growing adoption of Retrieval-Augmented Generation (RAG) in document
processing, robust text recognition has become increasingly critical for
knowledge extraction. While OCR (Optical Character Recognition) for English and
other languages benefits from large datasets and well-established benchmarks,
Arabic OCR faces unique challenges due to its cursive script, right-to-left
text flow, and complex typographic and calligraphic features. We present
KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in
current evaluation systems. Our benchmark comprises 8,809 samples across 9
major domains and 36 sub-domains, encompassing diverse document types including
handwritten text, structured tables, and specialized coverage of 21 chart types
for business intelligence. Our findings show that modern vision-language models
(such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like
EasyOCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate
(CER). Furthermore, we highlight significant limitations of current Arabic OCR
models, particularly in PDF-to-Markdown conversion, where the best model
Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in
accurately recognizing Arabic text, including issues with complex fonts,
numeral recognition errors, word elongation, and table structure detection.
This work establishes a rigorous evaluation framework that can drive
improvements in Arabic document analysis methods and bridge the performance gap
with English OCR technologies.