OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
  Retrieval-Augmented Generation
Authors: name: Junyuan Zhang; name: Qintong Zhang; name: Bin Wang; name: Linke Ouyang; name: Zichen Wen; name: Ying Li; name: Ka-Ho Chow; name: Conghui He; name: Wentao Zhang
Published: 2024-12-03 17:23:47

Abstract: Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
integrating external knowledge to reduce hallucinations and incorporate
up-to-date information without retraining. As an essential part of RAG,
external knowledge bases are commonly built by extracting structured data from
unstructured PDF documents using Optical Character Recognition (OCR). However,
given the imperfect prediction of OCR and the inherent non-uniform
representation of structured data, knowledge bases inevitably contain various
OCR noises. In this paper, we introduce OHRBench, the first benchmark for
understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
carefully selected unstructured PDF documents from six real-world RAG
application domains, along with Q&As derived from multimodal elements in
documents, challenging existing OCR solutions used for RAG To better understand
OCR's impact on RAG systems, we identify two primary types of OCR noise:
Semantic Noise and Formatting Noise and apply perturbation to generate a set of
structured data with varying degrees of each OCR noise. Using OHRBench, we
first conduct a comprehensive evaluation of current OCR solutions and reveal
that none is competent for constructing high-quality knowledge bases for RAG
systems. We then systematically evaluate the impact of these two noise types
and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
potential of employing Vision-Language Models (VLMs) without OCR in RAG
systems. Code: https://github.com/opendatalab/OHR-Bench