# A Comparative Analysis of OCR Models on Diverse Datasets: Insights from Memes and Hiertext Dataset

Iknoor Singh, Miguel Colom, Kalina Bontcheva

► To cite this version:

Iknoor Singh, Miguel Colom, Kalina Bontcheva. A Comparative Analysis of OCR Models on Diverse Datasets: Insights from Memes and Hiertext Dataset. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Feb 2025, Tucson, United States. pp.1343-1353, 10.1109/wacvw65960.2025.00148 . hal-05482382

HAL Id: hal-05482382

https://hal.science/hal-05482382v1

Submitted on 6 Feb 2026

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# A Comparative Analysis of OCR Models on Diverse Datasets: Insights from Memes and Hiertext Dataset

Iknoor Singh

University of Sheffield, UK

i.singh@sheffield.ac.uk

Miguel Colom

Centre Borelli, ENS Paris-Saclay

miguel.colom-barco@ens-paris-saclay.fr

Kalina Bontcheva

University of Sheffield, UK

k.bontcheva@sheffield.ac.uk

## Abstract

Optical Character Recognition (OCR) plays a critical role in various text extraction applications, yet its performance varies significantly across languages, visual appearances, and document types. This study benchmarks several state-of-the-art OCR models on two distinct and challenging datasets: MemeDataset, which contains informal, meme-based images, and HierText, featuring multilingual text in natural scenes and documents. We comprehensively evaluate a range of OCR models on these datasets, highlighting their strengths and limitations. While large multimodal models excel in complex text extraction, they are hindered by slower processing times and irrelevant text generation in challenging scenarios. Also, we examine the models' ability to handle multilingual content, finding that some newer models perform well in certain languages but struggle in others. Our results emphasise the need for further enhancements in open-sourced large multimodal models to improve efficiency and accuracy, particularly in multilingual scenarios. In contrast, other open-source models such as EasyOCR deliver faster inference times with reasonably competitive performance, making them more suitable for time-sensitive applications. The findings offer valuable insights for researchers and practitioners in selecting the right models for diverse use cases.

## 1. Introduction

Much of the textual content found in our surroundings contains valuable semantic information, which is essential for various real-world applications. Optical Character Recognition (OCR) plays a pivotal role in numerous applications such as scene-text reading [50], document text extraction and visual data analysis [40]. Notable advances

have been made in text detection and recognition, even in challenging situations such as text in video [52], curved format [8, 54], or images with multi-lingual content [41, 42].

Recently, the rapid development of Large Language Models (LLMs) like GPT-4¹ has unlocked remarkable applications in zero-shot learning across a wide range of real-world scenarios. The success of proprietary LLMs has fueled significant interest in open-source alternatives such as LLaMA [49] and Alpaca [48]. This success has also extended into the vision-language domain [6], as seen in the rise of large multimodal models like MiniCPM [51], Idefics [23], and Qwen-VL [5]. As shown in prior work [31], these multimodal models demonstrate impressive zero-shot OCR performance, even without specialised training on OCR-specific datasets. Despite notable advancements in OCR technologies, the task remains especially challenging for complex datasets such as multilingual memes and natural scene-text images. These datasets frequently feature text in diverse fonts, sizes, orientations, and languages, presenting a demanding testbed for OCR models. Therefore, identifying these challenges is essential to foster the development of more robust models capable of recognising text in challenging scenarios.

To this end, we evaluate the performance of several state-of-the-art end-to-end OCR models on two challenging datasets: the HierText dataset [34, 35], which provides hierarchical annotations from natural scenes and documents, and a multilingual meme dataset from the SemEval 2024 shared task [9]. Through comprehensive benchmarking, we highlight the strengths and weaknesses of each model, identifying Google Vision and GPT-4o-mini as top performers across both datasets. Additionally, we observe that open-source models like EasyOCR offer competitive results de-

¹https://openai.com/index/gpt-4/
spite being outperformed by closed-source models. Meanwhile, large multimodal models struggle with blurry and multilingual text, often generating irrelevant text. This underscores a promising opportunity to enhance large multimodal models' OCR capabilities through domain-specific training and adaptation. This study offers valuable insights into the current strengths and limitations of OCR systems, with a focus on real-world, multilingual, and complex text extraction tasks.

## 2. Related Work

In recent years, a variety of OCR models have emerged, ranging from traditional open-source models like Tesseract [47] to more advanced large multimodal models [5, 31, 51]. OCR typically relies on two-stage architectures to handle both detection and recognition tasks. Numerous deep learning-based methods have been proposed in the literature [2, 4, 12, 25, 45]. However, the performance of these models can vary significantly depending on dataset characteristics, such as language, text structure, and visual complexity. Moreover, the effectiveness of deep learning in text recognition tasks is largely dependent on the availability of extensive labelled datasets. For instance, a common strategy in text recognition [2, 12, 37] involves training models on large-scale synthetic datasets [14, 17, 18] and evaluating their performance on various benchmarks [20, 21, 39, 43, 44]. However, models trained on synthetic data often struggle with generalisation due to the significant domain gap between synthetic datasets and real-world scenarios [3]. This study aims to evaluate the robustness of text recognition models when faced with real-world challenges, identify current limitations, and promote further advancements in the field.

Furthermore, the success of LLMs has inspired the creation of large multimodal models, which integrate learning from both textual and visual information to extend the capabilities of models on various tasks. For instance, Qwen-VL [5], developed by Alibaba, excels in visual understanding and generation tasks, showcasing strong performance across various benchmarks. LLaVA [30] is one of the first to use GPT-4 for generating multimodal instruction-following data for training the model. Monkey [26], on the other hand, focuses on improving image resolution processing and text labelling, leading to enhanced multimodal capabilities. These models also tend to rely heavily on semantic understanding, favouring common words while generation [33]. Other research efforts have also focused on better aligning vision modules with LLMs to enhance multimodal understanding [29, 55]. Microsoft's Phi-3 Vision [1] features a large 128K token context window, offering enhanced capabilities for vision-language tasks, particularly in long-context scenarios. MiniCPM [51], a compact yet efficient model, is designed for deployment in resource-constrained environments while maintaining competitive performance.

Additionally, InternLM XComposer2 [10] emphasises free-form text-image composition and comprehension with its 7-billion parameter architecture. Lastly, mPLUG DocOwl 1.5 [16] specialises in document understanding, excelling in tasks that involve complex layouts and the extraction of structured information from documents.

As large multimodal models continue to advance, the challenge of evaluating their performance has become increasingly important. Prior work [11, 32, 53] provides effective frameworks for assessing these models. MME [13] measures both the perception and cognition abilities of the models, while MMBench [32] uses multiple-choice questions to evaluate large multimodal models across several dimensions. However, these evaluations have only minimally addressed the specific challenges presented while doing OCR tasks. On the other hand, [33] proposed OCRBench to evaluate OCR capabilities in large multimodal models. Nonetheless, they did not include meme-style datasets or the new HierText dataset in their evaluation [33]. Additionally, their work primarily focused on large multimodal models, whereas our interest lies in comparing how the new transformer-based multimodal models perform relative to the existing non-transformer-based models.

In this paper, we conduct an extensive zero-shot evaluation of OCR models on the SemEval 2024 Task 4 Meme dataset, as well as the new HierText scene-text dataset. Both datasets present unique challenges i.e. the Meme dataset tests the models' ability to extract text from images with complex layouts, fonts, and background noise, while the HierText dataset assesses performance on text in real-world environments.

## 3. Experimental setup

### 3.1. Evaluation datasets

To evaluate the performance of various OCR models, we utilised two distinct datasets:

* **Memes Dataset** [9]: This dataset is part of the SemEval 2024 shared task 4 on "Multilingual Detection of Persuasion Techniques in Memes." It consists of meme images in multiple languages: English (EN), Bulgarian (BG), Arabic (AR), and North Macedonian (MD). The dataset provides an opportunity to assess how well OCR models can extract text from meme-style images, which often feature challenging layouts, fonts, and background clutter. We compare the text extracted by OCR models with the actual text annotations provided in the dataset. For testing, we use the validation set of the Memes Dataset, which includes 500 images in English, 160 images in Arabic, and 100 images each in Bulgarian and North Macedonian.
<figure><img src="image_2.png" /><figcaption>Figure 1. Sample images from the Memes Dataset [9] and the HierText Dataset [34, 35].</figcaption></figure>

* **HierText Dataset** [34,35]: HierText is the first dataset to provide hierarchical annotations of text in both natural scenes and documents. The dataset consists of images from the Open Images dataset and includes high-quality annotations at the word, line, and paragraph levels. Due to the complexity of real-world scene-level images and the order in which the gold annotations were provided, we could not use the complete image for inference. Therefore, to evaluate the OCR models on this dataset, we cropped line-level bounding boxes from the test images and sampled 1,000 cropped images for testing. On average, each line-level cropped image contains four words. We test different text recognition models on the cropped images to assess how well they handle text in natural scenes and complex document structures.

## 3.2. OCR Models

We tested a wide range of OCR models on the Meme and Hiertext datasets. These are as follows:

* **Tesseract** [47] is an open-source OCR engine originally developed by Hewlett-Packard and currently maintained by Google. It uses a long short-term memory (LSTM) architecture to recognise text. We use PyTesseract<fnref n="2" /> for inference.

* **EasyOCR<fnref n="3" />** is an open-source Python library, which provides scene text recognition (STR) models for multiple language scripts. These models use a Convolutional-Recurrent Neural Network (CRNN) [2, 45] architecture, incorporating ResNet50 [15] for feature extraction and bidirectional LSTM for sequence labelling. For text detection, the Character Region Awareness For Text (CRAFT) algorithm [4,46] is used to segment input images and retain only the areas containing text. EasyOCR has multi-language support and is designed to recognise text from a wide range of images.

* **PaddleOCR<fnref n="4" />** is an open-source OCR library created by PaddlePaddle, a leading AI research and development

<footnote n="2">²https://pypi.org/project/pytesseract/</footnote>

<footnote n="3">³https://github.com/JaidedAI/EasyOCR</footnote>

<footnote n="4">⁴https://github.com/PaddlePaddle/PaddleOCR</footnote>
ment organisation. It provides various multilingual models based on PP-OCRv3 [25] architecture.

* **Document Text Recognition (DocTR)** [38] is an open-source deep learning-based OCR library that focuses on text detection and recognition in document images. For detection, DocTR uses a base version of faster arbitrarily shaped text detector using minimalist kernel representation [7], while its recognition model is a CRNN model with a VGG-16 backbone [45].

* **KerasOCR<fnref n="5" />** is an easy-to-use OCR pipeline built with Keras, consisting of two main components: text detection using the custom version of the CRAFT model [4] and text recognition using a CRNN model [45].

* **MMOCR<fnref n="6" />** [22] is an open-source library created by OpenMMLab that provides out-of-the-box OCR models for both text detection and recognition. We evaluate the following text detection models: 1) DBNetpp [28], an improved version of DBNet (Differentiable Binarization Network) [27], designed for robust text detection; 2) TextSnake [36], a model developed to handle curved and irregular text. For text recognition, we assess: 1) ABINet (Autonomous, Bidirectional, and Iterative Language Modeling) [12], which uses language modelling to enhance accuracy; 2) SATRN [24], which leverages self-attention mechanisms to capture long-range dependencies in the text; and 3) MASTER (Multi-Aspect Non-local Network for Scene Text Recognition) [37], self-attention based scene text recogniser built to handle text in various fonts, sizes, and orientations.

* **Google Vision OCR<fnref n="7" />** is part of Google Cloud's Vision AI services. It leverages Google's proprietary deep learning models to perform OCR on images, supporting a wide range of languages and text types.

* **Open-Source Large Multimodal Models:** We test several open-source large multimodal models that can perform OCR tasks among other vision-language capabilities. These models are trained on large datasets of images and text, allowing them to understand and process visual information alongside text. While not specifically designed for OCR, they can often perform OCR tasks effectively due to their broad understanding of visual and textual data. We use the following prompt to recognise text: "Recognise the text in this image and return only the text as a string." The models used are as follows: 1) *Qwen-VL* [5], developed by Alibaba, which excels in visual understanding and gener-

ation tasks; 2) *Monkey* [26], which focuses on improving image resolution processing and text labelling for enhanced multimodal performance; 3) *Ideics2* [23], an 8B parameter model by Hugging Face known for its strong zero-shot capabilities across various tasks; 4) *Phi-3-Vision-128K-Instruct* [1], Microsoft's model featuring a large 128K token context window, likely with enhanced capabilities for vision-language tasks; 5) *MiniCPM-v2* [51], a compact yet powerful model designed for efficient deployment while maintaining competitive performance; 6) *InternLM-XComposer2-vl-7b* [10], a 7B parameter model focusing on free-form text-image composition and comprehension; 7) *LLaVA-v1.6-Mistral-7b* [19], an updated version of the LLaVA model [30], based on the Mistral-7B language model architecture; and 8) *mPLUG-DocOwl-1.5* [16], which specialises in document understanding and is capable of handling complex layouts and extracting structured information. We also test the multilingual OCR capabilities of the *Qwen-VL* and *MiniCPM-v2*.

* **Closed-Source Large Multimodal Models:** We also test the closed-source large multimodal models: 1) *GPT-4*<fnref n="8" />, OpenAI's vision-language model is used to check OCR functionality as part of its multimodal capabilities. For our analysis, we evaluate the *GPT-4o-mini* model. 2) *Gemini-1.5-pro*<fnref n="9" />, a multimodal AI model by Google. Both GPT-4 and Gemini-1.5 can handle a wide range of text types and languages, making them suitable for multilingual OCR tasks.

### 3.3. Evaluation metrics

To evaluate the performance of the OCR models, we use three different metrics to assess the quality of text recognition at both the character and word levels. These metrics are as follows,

* **Character Accuracy:** Character Error Rate (CER) measures the ratio of character-level errors (insertions, deletions, substitutions) to the total characters in the ground truth. It is calculated as:

$$CER = \frac{\text{Character-level Levenshtein Distance}}{\text{Total Characters}}$$

Character Accuracy is 1 - CER, with higher values indicating better character recognition.

* **Word Accuracy:** Word Error Rate (WER) assesses the ratio of word-level errors to the total words in the ground truth, computed as:

$$WER = \frac{\text{Word-level Levenshtein Distance}}{\text{Total Words}}$$

<footnote n="5">⁵https://github.com/faustomorales/keras-ocr</footnote>

<footnote n="6">⁶https://github.com/open-mmlab/mmocr</footnote>

<footnote n="7">⁷https://cloud.google.com/vision</footnote>

<footnote n="8">⁸https://openai.com/index/gpt-4/</footnote>

<footnote n="9">⁹https://deepmind.google/technologies/gemini/</footnote>
<table><caption>Table 1. OCR benchmark results on English Memes dataset. The best results are in bold.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Case sensitive</th><th colspan="3">Case insensitive</th><th rowspan="2">Time</th></tr><tr><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th></tr></thead><tbody><tr><td>EasyOCR</td><td>0.4615</td><td>0.2891</td><td>0.2864</td><td>0.4739</td><td>0.3063</td><td>0.3083</td><td><strong>0.1938</strong></td></tr><tr><td>PaddleOCR</td><td>0.4656</td><td>0.2736</td><td>0.2851</td><td>0.4698</td><td>0.2777</td><td>0.2896</td><td>0.8069</td></tr><tr><td>Tesseract</td><td>0.2578</td><td>0.0486</td><td>0.0218</td><td>0.2734</td><td>0.0517</td><td>0.0319</td><td>0.7330</td></tr><tr><td>DocTR</td><td>0.2783</td><td>-0.0776</td><td>-0.0748</td><td>0.2867</td><td>-0.0679</td><td>-0.0600</td><td>6.0000</td></tr><tr><td>Dbnetpp+ABINet</td><td>-0.1798</td><td>-0.4171</td><td>-0.3660</td><td>0.0669</td><td>-0.2197</td><td>0.0301</td><td>0.6000</td></tr><tr><td>TextSnake+ABInet</td><td>-0.0720</td><td>-0.1508</td><td>-0.1506</td><td>0.2060</td><td>-0.1120</td><td>-0.1039</td><td>0.4400</td></tr><tr><td>DBNetpp+SATRN</td><td>-0.0159</td><td>-0.2170</td><td>-0.0069</td><td>0.0091</td><td>-0.1982</td><td>0.0312</td><td>3.0000</td></tr><tr><td>DBNetpp+MASTER</td><td>-0.0029</td><td>-0.2174</td><td>0.0044</td><td>0.0209</td><td>-0.1982</td><td>0.0418</td><td>1.2000</td></tr><tr><td>Keras OCR</td><td>0.1964</td><td>-0.0531</td><td>0.2167</td><td>0.1964</td><td>-0.0531</td><td>0.2167</td><td>0.6000</td></tr><tr><td>mPLUG-DocOwl-1.5</td><td>0.2709</td><td>0.1977</td><td>0.1840</td><td>0.3883</td><td>0.3393</td><td>0.3352</td><td>0.9071</td></tr><tr><td>Qwen-VL</td><td>0.3656</td><td>0.2387</td><td>0.2071</td><td>0.6255</td><td>0.5698</td><td>0.5674</td><td>0.5120</td></tr><tr><td>Monkey</td><td>0.2952</td><td>0.2054</td><td>0.1855</td><td>0.4852</td><td>0.4390</td><td>0.4367</td><td>0.9100</td></tr><tr><td>Idefics2-8b</td><td>0.5680</td><td>0.4584</td><td>0.4393</td><td><strong>0.7894</strong></td><td><strong>0.7165</strong></td><td><strong>0.7193</strong></td><td>3.3368</td></tr><tr><td>Phi-3-Vision-128K-Instruct</td><td>0.5795</td><td>0.4319</td><td>0.4241</td><td>0.6042</td><td>0.4643</td><td>0.4650</td><td>1.8000</td></tr><tr><td>MiniCPM-v2</td><td>0.5466</td><td>0.4489</td><td>0.4397</td><td>0.6321</td><td>0.5526</td><td>0.5574</td><td>0.7224</td></tr><tr><td>InternLM-XComposer2-vl-7b</td><td>0.4806</td><td>0.3868</td><td>0.3734</td><td>0.7457</td><td>0.7056</td><td>0.7104</td><td>3.3000</td></tr><tr><td>LLaVA-v1.6-Mistral-7b</td><td>0.4643</td><td>0.3751</td><td>0.3571</td><td>0.5909</td><td>0.5318</td><td>0.5324</td><td>1.5800</td></tr><tr><td>Gemini-1.5-pro</td><td>0.5817</td><td>0.5768</td><td>0.5813</td><td>0.5852</td><td>0.5820</td><td>0.5889</td><td>2.2000</td></tr><tr><td>GPT-4o-mini</td><td><strong>0.6994</strong></td><td><strong>0.6790</strong></td><td><strong>0.6768</strong></td><td>0.7098</td><td>0.6926</td><td>0.6931</td><td>3.6000</td></tr><tr><td>Google Vision</td><td>0.4175</td><td>0.4074</td><td>0.4180</td><td>0.4217</td><td>0.4134</td><td>0.4236</td><td>0.2776</td></tr></tbody></table>

<table><caption>Table 2. OCR benchmark results on Arabic Memes dataset. The best results are in bold.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Case sensitive</th><th colspan="3">Case insensitive</th></tr><tr><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th></tr></thead><tbody><tr><td>EasyOCR</td><td>0.6299</td><td>0.3731</td><td>0.3590</td><td>0.6299</td><td>0.3731</td><td>0.3590</td></tr><tr><td>PaddleOCR</td><td>0.0572</td><td>-0.2018</td><td>-0.1983</td><td>0.0572</td><td>-0.2018</td><td>-0.1983</td></tr><tr><td>MiniCPM-v2</td><td>0.1197</td><td>0.0032</td><td>0.0049</td><td>0.1206</td><td>0.0047</td><td>0.0042</td></tr><tr><td>Qwen-VL</td><td>-0.9445</td><td>-0.7711</td><td>-0.7711</td><td>-0.9445</td><td>-0.7711</td><td>-0.7711</td></tr><tr><td>GPT-4o-mini</td><td><strong>0.8110</strong></td><td><strong>0.5605</strong></td><td><strong>0.5523</strong></td><td><strong>0.8110</strong></td><td><strong>0.5605</strong></td><td><strong>0.5523</strong></td></tr><tr><td>Google Vision</td><td>0.5911</td><td>0.4702</td><td>0.4921</td><td>0.5911</td><td>0.4702</td><td>0.4921</td></tr></tbody></table>

Word Accuracy is 1 - WER, evaluating the overall word recognition performance.

* **Position Independent Word Accuracy:** Position Independent Word Error Rate (PI - WER) measures word error rate without considering word order. Position Independent Word Accuracy is calculated as 1 - PI - WER, which is useful for scenarios where one does not want to penalise wrong word order.

These metrics provide a comprehensive evaluation of character and word recognition. Please note that both character and word accuracy can be negative when the number of errors exceeds the total number of characters or words in the reference, indicating particularly poor recognition performance.

# 4. Results and Discussion

## 4.1. Memes Dataset

Table 1 presents the OCR benchmark results for various models tested on the English subset of the Meme dataset. The performance is evaluated using three key metrics: Character Accuracy (Char Acc), Word Accuracy (Word Acc), and Position Independent Word Accuracy (PI-Word Acc), with results shown for both case-sensitive and case-insensitive settings. Additionally, we report the processing time for each model.

The results indicate significant variability in performance across the different models. Text recognition models like ABInet (Dbnetpp+ABInet and TextSnake+ABInet) and SATRN (DBNetpp+SATRN) consistently show lower
<table><caption>Table 3. OCR benchmark results on Bulgarian Memes dataset. The best results are in bold.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Case sensitive</th><th colspan="3">Case insensitive</th></tr><tr><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th></tr></thead><tbody><tr><td>EasyOCR</td><td>0.4891</td><td>0.2737</td><td>0.2541</td><td>0.5962</td><td>0.3514</td><td>0.3388</td></tr><tr><td>PaddleOCR</td><td>0.4820</td><td>0.1378</td><td>0.0852</td><td>0.5782</td><td>0.1874</td><td>0.1245</td></tr><tr><td>MiniCPM-v2</td><td>0.1838</td><td>0.0471</td><td>0.0448</td><td>0.2121</td><td>0.0483</td><td>0.0448</td></tr><tr><td>Qwen-VL</td><td>-0.0491</td><td>-0.1739</td><td>-0.1769</td><td>-0.0421</td><td>-0.1739</td><td>-0.1777</td></tr><tr><td>GPT-4o-mini</td><td><strong>0.8109</strong></td><td><strong>0.7290</strong></td><td><strong>0.7025</strong></td><td><strong>0.8385</strong></td><td><strong>0.7671</strong></td><td><strong>0.7546</strong></td></tr><tr><td>Google Vision</td><td>0.6240</td><td>0.5750</td><td>0.5756</td><td>0.6329</td><td>0.5947</td><td>0.6113</td></tr></tbody></table>

<table><caption>Table 4. OCR benchmark results on North Macedonian Memes dataset. The best results are in bold.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Case sensitive</th><th colspan="3">Case insensitive</th></tr><tr><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th></tr></thead><tbody><tr><td>EasyOCR</td><td>0.1728</td><td>-0.0072</td><td>-0.0333</td><td>0.3483</td><td>0.1167</td><td>0.1020</td></tr><tr><td>PaddleOCR</td><td>0.1479</td><td>-0.0628</td><td>-0.0919</td><td>0.3446</td><td>0.0271</td><td>-0.0295</td></tr><tr><td>MiniCPM-v2</td><td>0.1759</td><td>0.0295</td><td>0.0320</td><td>0.2147</td><td>0.0365</td><td>0.0373</td></tr><tr><td>Qwen-VL</td><td>-0.5075</td><td>-0.0670</td><td>-0.0670</td><td>-0.4754</td><td>-0.0507</td><td>-0.0499</td></tr><tr><td>GPT-4o-mini</td><td><strong>0.7167</strong></td><td><strong>0.6150</strong></td><td><strong>0.5868</strong></td><td><strong>0.8103</strong></td><td><strong>0.7216</strong></td><td><strong>0.6989</strong></td></tr><tr><td>Google Vision</td><td>0.4232</td><td>0.3293</td><td>0.3271</td><td>0.4338</td><td>0.3527</td><td>0.3573</td></tr></tbody></table>

scores. In contrast, Tesseract achieves a character accuracy of 0.2578 in case-sensitive mode, but its word accuracy drops drastically to 0.0486, highlighting its difficulty in handling complete words.

Modern models like EasyOCR and PaddleOCR perform more consistently across both character and word-level metrics. EasyOCR, for example, achieves a balanced performance with a character accuracy of 0.4615 and a word accuracy of 0.2891 in the case-sensitive setting. These models also tend to have lower processing times, indicating their efficiency in handling OCR tasks on meme-type images.

Open-source large multimodal models, such as Idefics2 and MiniCPM-v2, exhibit strong performance in both case-sensitive and case-insensitive evaluations. Idefics2, for instance, achieves a character accuracy of 0.5680 in case-sensitive mode, which improves to 0.7894 in case-insensitive mode, demonstrating its ability to adapt to various text formats and cases. However, these models generally have higher processing times, with Idefics2 taking 3.3368 seconds per image. This can be a drawback for large-scale applications.

Among the closed-source large multimodal models, GPT-4o-mini shows the highest accuracy across the board, with character and word accuracies of 0.6994 and 0.6790, respectively, in case-sensitive mode. Interestingly, GPT-4o-mini performs even better than the Google Vision API. Their performance further improves in the case-insensitive setting, suggesting that they are capable of handling OCR

tasks effectively, even when confronted with diverse text styles and cases.

Table 2, 3 & 4 present OCR benchmark results for multilingual OCR models across the Arabic, Bulgarian, and North Macedonian meme datasets, respectively. The results show variability in model performance, with GPT-4o-mini consistently outperforming other models across all three languages. For instance, it achieves the highest character and word accuracies, such as a word accuracy of 0.5605 in Arabic and 0.7290 in Bulgarian. This demonstrates its strong ability to handle diverse scripts and complex text styles. Google Vision also performs well, particularly in Bulgarian, but trails behind GPT-4o-mini in most metrics.

Among open-source models, EasyOCR stands out as the best performer, delivering competitive results across all datasets. For example, on the Arabic dataset, EasyOCR achieves a word accuracy of 0.3731, surpassing other open-source multilingual models like PaddleOCR and MiniCPM-v2. This highlights EasyOCR's efficiency in handling OCR tasks in multilingual, meme-style images, although it still falls short of the closed-source GPT-4o-mini and Google Vision in overall accuracy.

Overall, whereas OCR models like EasyOCR and PaddleOCR provide decent baseline performance, more modern models and large multimodal systems like GPT-4o-mini and Idefics2 demonstrate superior results, particularly in handling complex, irregular, and multilingual text found in meme-type images. The choice of OCR model depends on
<table><caption>Table 5. OCR benchmark results on HierText dataset. The best results are in bold.</caption><thead><tr><th rowspan="2">Models</th><th colspan="3">Case sensitive</th><th colspan="3">Case insensitive</th><th rowspan="2">Time</th></tr><tr><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th><th>Char Acc</th><th>Word Acc</th><th>PI-Word Acc</th></tr></thead><tbody><tr><td>EasyOCR</td><td>0.4162</td><td>0.2207</td><td>0.2172</td><td>0.4457</td><td>0.2412</td><td>0.2390</td><td><strong>0.0420</strong></td></tr><tr><td>PaddleOCR</td><td>0.3838</td><td>0.1638</td><td>0.1542</td><td>0.3858</td><td>0.1656</td><td>0.1576</td><td>0.1100</td></tr><tr><td>Tesseract</td><td>0.3634</td><td>0.2262</td><td>0.2149</td><td>0.3712</td><td>0.2318</td><td>0.2214</td><td>0.1900</td></tr><tr><td>ABINet</td><td>0.3274</td><td>0.0045</td><td>0.0045</td><td>0.5416</td><td>0.0070</td><td>0.0070</td><td>0.0700</td></tr><tr><td>SATRN</td><td>0.4862</td><td>0.0025</td><td>0.0025</td><td>0.5479</td><td>0.0025</td><td>0.0025</td><td>0.3600</td></tr><tr><td>MASTER</td><td>0.4888</td><td>0.0005</td><td>0.0005</td><td>0.5410</td><td>0.0005</td><td>0.0005</td><td>0.2700</td></tr><tr><td>KerasOCR</td><td>0.3476</td><td>0.0649</td><td>0.2197</td><td>0.3476</td><td>0.0649</td><td>0.2197</td><td>0.3400</td></tr><tr><td>Monkey</td><td>-0.5251</td><td>-0.7472</td><td>-0.7513</td><td>-0.4356</td><td>-0.6932</td><td>-0.6976</td><td>0.7500</td></tr><tr><td>Idefics2-8b</td><td>0.2815</td><td>0.0331</td><td>0.0089</td><td>0.3423</td><td>0.0956</td><td>0.0852</td><td>1.0500</td></tr><tr><td>Phi-3-Vision-128k-Instruct</td><td>-1.2023</td><td>-1.1258</td><td>-1.1328</td><td>-1.0325</td><td>-1.0004</td><td>-1.0020</td><td>1.0000</td></tr><tr><td>MiniCPM-v2</td><td>0.6677</td><td>0.1469</td><td>0.1435</td><td>0.6828</td><td>0.1539</td><td>0.1509</td><td>0.2550</td></tr><tr><td>InternLM-XComposer2-vl-7b</td><td>-0.7128</td><td>-0.5100</td><td>-0.5199</td><td>-0.5895</td><td>-0.4016</td><td>-0.4060</td><td>2.7000</td></tr><tr><td>LLaVA-v1.6-Mistral-7b</td><td>-1.9409</td><td>-1.9976</td><td>-1.9993</td><td>-1.8073</td><td>-1.8716</td><td>-1.8696</td><td>1.0300</td></tr><tr><td>Gemini-1.5-pro</td><td>0.3456</td><td>0.3076</td><td>0.3012</td><td>0.3674</td><td>0.3210</td><td>0.3158</td><td>2.9000</td></tr><tr><td>GPT-4o-mini</td><td>0.2655</td><td>0.1480</td><td>0.1361</td><td>0.2969</td><td>0.1606</td><td>0.1506</td><td>1.5800</td></tr><tr><td>Google Vision</td><td><strong>0.7580</strong></td><td><strong>0.6256</strong></td><td><strong>0.6197</strong></td><td><strong>0.7607</strong></td><td><strong>0.6326</strong></td><td><strong>0.6287</strong></td><td>0.1500</td></tr></tbody></table>

the trade-off between accuracy and processing time, with vision-language models offering high accuracy at the cost of increased inference time.

## 4.2. HierText Dataset

The OCR benchmark results on the HierText dataset (Table 5) reveal notable differences in model performance. Google Vision leads the results, achieving the highest character and word accuracies, with a word accuracy of 0.6256 in the case-sensitive setting and 0.6326 in the case-insensitive setting. In contrast, large multimodal models like Monkey, Phi-3-Vision-128K-Instruct, and LLaVA-v1.6-Mistral-7b display negative accuracies, indicating severe underperformance on this scene-text dataset. This underperformance stems from the models generating irrelevant text, such as “Sorry, but I cannot recognise any text due to a blurred image” or “No text is present in the image.” Despite attempts to explicitly instruct the models not to generate such text and return an empty string instead, they consistently failed to comply.

Other models, such as Gemini-1.5-pro and EasyOCR, show competitive performance, with Gemini-1.5-pro attaining a word accuracy of 0.3076 and EasyOCR achieving 0.2207 in the case-sensitive setting. Several open-source models, like ABINet, SATRN, and MASTER, struggle significantly, with word accuracies close to zero, despite having relatively high character accuracy. This is because these models tend to correctly recognise individual characters but struggle with word-level accuracy due to poor handling of complex scene text structures or inconsistencies in word segmentation. Additionally, we found that models often

merge multiple words into one, ignoring the spaces that separate individual words.

Regarding inference speed, similar to the Meme dataset (Table 1), large multimodal models like Monkey and LLaVA-v1.6-Mistral-7b take significantly longer, with times of 0.7500 and 1.0300 seconds, respectively. In contrast, open-source models such as EasyOCR and PaddleOCR offer faster processing times of 0.0420 and 0.1100 seconds, making them more suitable for time-sensitive applications.

## 4.3. Efficiency Analysis

We assess the efficiency of various OCR models, which are defined as character accuracy (case-insensitive) divided by the inference time taken per image. Higher efficiency values indicate that a model can perform accurate OCR quickly, making it more suitable for real-world applications where both accuracy and speed are critical. All the open-sourced models are tested on NVIDIA A100 cards, except for PaddleOCR, which could not be run on GPU due to the complexity involved. For closed-source models, we used their respective APIs for evaluation.

Figure 2 illustrates the efficiency results for the models evaluated on the English subset of the Meme dataset. It shows that EasyOCR emerges as the most efficient model with a score of 2.4454, followed by Google Vision with an efficiency of 1.5191 and Qwen-VL at 1.2217. These models demonstrate a strong balance between character accuracy and inference speed, making them suitable for high-demand environments. On the other hand, large multimodal models like Qwen-VL and MiniCPM-v2 also show competitive
<figure><img src="image_3.png" /><figcaption>Figure 2. Efficiency analysis of OCR models. The Y-axis represents efficiency, defined as case-insensitive character accuracy divided by the inference time per image.</figcaption></figure>

efficiency values of 1.2217 and 0.8750, respectively, indicating their capacity to handle OCR tasks effectively while maintaining moderate speed.

Overall, the efficiency metric offers valuable insights into the trade-offs between speed and accuracy in OCR tasks. The results show that while advanced multimodal models are improving, other OCR models like EasyOCR remain more efficient for text extraction tasks. These findings suggest an opportunity for optimising the speed and accuracy of multimodal models to make them more practical for real-world OCR applications.

## 5. Conclusion

In this study, we comprehensively evaluated a range of OCR models on two distinct datasets, MemeDataset and HierText, highlighting their strengths and limitations in extracting text from multilingual images and visually diverse document types. While closed-source GPT-4o-mini and Google Vision delivered top accuracy, open-source models such as EasyOCR provided a strong balance between accuracy and speed. This makes EasyOCR more suitable for time-sensitive and cost-effective applications. We also find that the open-sourced large multimodal models struggled with complex text, often generating irrelevant text. Overall, traditional OCR models remain more efficient, while large multimodal models offer potential but need further optimization for real-world use.

## Acknowledgements

This work has been co-funded by the UK's innovation agency (Innovate UK) grant 10039055 (approved under the

Horizon Europe Programme as vera.ai, EU grant agreement 101070093) under action number 2020-EU-IA-0282.

## References

[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. ArXiv preprint, abs/2404.14219, 2024. 2, 4

[2] Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4714-4722. IEEE, 2019. 2, 3

[3] Jeonghun Baek, Yusuke Matsui, and Kiyoharu Aizawa. What if we only use real datasets for scene text recognition? toward scene text recognition with fewer labels. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3113-3122. Computer Vision Foundation / IEEE, 2021. 2

[4] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. Character region awareness for text detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9365-9374. Computer Vision Foundation / IEEE, 2019. 2, 3, 4

[5] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-v1: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023. 1, 2, 4
[6] Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. *Machine Intelligence Research*, 20(1):38–56, 2023. 1

[7] Zhe Chen, Jiahao Wang, Wenhai Wang, Guo Chen, Enze Xie, Ping Luo, and Tong Lu. Fast: Faster arbitrarily-shaped text detector with minimalist kernel representation. *ArXiv preprint*, abs/2111.02394, 2021. 4

[8] Chee Kheng Ch'ng and Chee Seng Chan. Total-text: A comprehensive dataset for scene text detection and recognition. In *2017 14th IAPR international conference on document analysis and recognition (ICDAR)*, volume 1, pages 935–942. IEEE, 2017. 1

[9] Dimitar Dimitrov, Firoj Alam, Maram Hasanain, Abul Hasnat, Fabrizio Silvestri, Preslav Nakov, and Giovanni Da San Martino. Semeval-2024 task 4: Multilingual detection of persuasion techniques in memes. In *Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)*, pages 2009–2026, 2024. 1, 2, 3

[10] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. *ArXiv preprint*, abs/2401.16420, 2024. 2, 4

[11] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. *ArXiv preprint*, abs/2407.11691, 2024. 2

[12] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In *IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19–25, 2021*, pages 7098–7107. Computer Vision Foundation / IEEE, 2021. 2, 4

[13] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023. 2

[14] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In *2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016*, pages 2315–2324. IEEE Computer Society, 2016. 2

[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016*, pages 770–778. IEEE Computer Society, 2016. 3

[16] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. *ArXiv preprint*, abs/2403.12895, 2024. 2, 4

[17] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. *arXiv preprint arXiv:1406.2227*, 2014. 2

[18] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. *International journal of computer vision*, 116:1–20, 2016. 2

[19] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lampe, Lucile Saulnier, et al. Mistral 7b. *ArXiv preprint*, abs/2310.06825, 2023. 4

[20] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In *2015 13th international conference on document analysis and recognition (ICDAR)*, pages 1156–1160. IEEE, 2015. 2

[21] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In *2013 12th international conference on document analysis and recognition*, pages 1484–1493. IEEE, 2013. 2

[22] Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, et al. Mmocr: a comprehensive toolbox for text detection, recognition and understanding. In *Proceedings of the 29th ACM International Conference on Multimedia*, pages 3791–3794, 2021. 4

[23] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? *ArXiv preprint*, abs/2405.02246, 2024. 1, 4

[24] Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, and Hwalsuk Lee. On recognizing texts of arbitrary shapes with 2d self-attention. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, pages 546–547, 2020. 4

[25] Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, et al. Pp-ocrv3: More attempts for the improvement of ultra lightweight ocr system. *ArXiv preprint*, abs/2206.03001, 2022. 2, 4

[26] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 26763–26773, 2024. 2, 4

[27] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In *The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020*,
The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11474–11481. AAAI Press, 2020. 4

[28] Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE transactions on pattern analysis and machine intelligence, 45(1):919–931, 2022. 4

[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2

[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023. 2, 4

[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 2

[32] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? ArXiv preprint, abs/2307.06281, 2023. 2

[33] Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. ArXiv preprint, abs/2305.07895, 2023. 2

[34] Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Towards end-to-end unified scene text detection and layout analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 1, 3

[35] Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. Icdar 2023 competition on hierarchical text detection and recognition. ArXiv preprint, abs/2305.09750, 2023. 1, 3

[36] Shangbang Long, Jiaqiang Ruan, Wenjie Zhang, Xin He, Wenhao Wu, and Cong Yao. Textsnake: A flexible representation for detecting text of arbitrary shapes. In Proceedings of the European conference on computer vision (ECCV), pages 20–36, 2018. 4

[37] Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, and Xiang Bai. Master: Multi-aspect non-local network for scene text recognition. Pattern Recognition, 117:107980, 2021. 2, 4

[38] Mindee. doctr: Document text recognition. https://github.com/mindee/doctr, 2021. 4

[39] Anand Mishra, Karteek Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In Richard Bowden, John P. Collomosse, and Krystian Mikolajczyk, editors, British Machine Vision Conference, BMVC 2012, Surrey, UK, September 3-7, 2012, pages 1–11. BMVA Press, 2012. 2

[40] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019. 1

[41] Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al. Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In 2019 International conference on document analysis and recognition (ICDAR), pages 1582–1587. IEEE, 2019. 1

[42] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, et al. Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt. In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1454–1459. IEEE, 2017. 1

[43] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pages 569–576. IEEE Computer Society, 2013. 2

[44] Anhar Risnumawan, Palaiahankote Shivakumara, Chee Seng Chan, and Chew Lim Tan. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, 41(18):8027–8048, 2014. 2

[45] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304, 2016. 2, 3, 4

[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 3

[47] Ray Smith. An overview of the tesseract ocr engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629–633. IEEE, 2007. 2, 3

[48] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023. 1

[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302.13971, 2023. 1

[50] Kai Wang, Boris Babenko, and Serge J. Belongie. End-to-end scene text recognition. In Dimitris N. Metaxas, Long Quan, Alberto Sanfeliu, and Luc Van Gool, editors, IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pages 1457–1464. IEEE Computer Society, 2011. 1

[51] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye
Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level mllm on your phone. ArXiv preprint, abs/2408.01800, 2024. 1, 2, 4

[52] Xu-Cheng Yin, Ze-Yu Zuo, Shu Tian, and Cheng-Lin Liu. Text detection, tracking and recognition in video: a comprehensive survey. IEEE Transactions on Image Processing, 25(6):2752-2773, 2016. 1

[53] Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. ArXiv preprint, abs/2404.16006, 2024. 2

[54] Liu Yuliang, Jin Lianwen, Zhang Shuaitao, and Zhang Sheng. Detecting curve text in the wild: New dataset and new solution. ArXiv preprint, abs/1712.02170, 2017. 1

[55] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv preprint, abs/2304.10592, 2023. 2