VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text
Authors: Tianyu Zhang
Suyuchen Wang
Lu Li
Ge Zhang
Perouz Taslakian
Sai Rajeswar
Jie Fu
Bang Liu
Yoshua Bengio
Venue: ICLR 2025 Poster (2025)
Keywords: dataset
vision-language model
multimodal
Primary area: datasets and benchmarks
TL;DR: We introduce a novel task: Visual Caption Restoration (VCR) along with a dataset which aims to challenge existing models by requiring them to restore text that is partially obscured within images.

Abstract: We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images through complex reasoning. This task stems from the observation that text embedded in images intrinsically differs from common visual elements and text due to the need to align the modalities of vision, text, and text embedded in images. While many works incorporate text into images for visual question answering, they mostly rely on OCR or masked language modeling, reducing the task to text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny, exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct VCR-WIKI for VCR using Wikipedia images with captions, including 2.11M English and 346K Chinese training entities, plus 5K validation and 5K test entities in both languages, each in easy and hard configurations. We also make a hidden test set, VCR-HIDDEN, to avoid potential overfitting on VCR-WIKI. Our results reveal that current vision-language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-WIKI and the data construction code to facilitate future research.

Published as a conference paper at ICLR 2025
VCR: A TASK FOR PIXEL-LEVEL COMPLEX REASONING IN VISION LANGUAGE MODELS VIA RESTORING OCCLUDED TEXT
Tianyu Zhangl.2." Suyuchen Wangl Lu Li", Ge Zhang Perouz Taslakian", Sai Rajeswar', Jie Fu', Bang Liu Yoshua Bengio Mila, Quebec AI Institute ServiccNow Research Université de Montréal
4 University of Pennsylvania University of Waterloo HKUST 7 CIFAR AI Chair ftianyu.zhand zhang, yoshua .oshua.bengio//e..o wewggio/emmaa.queee.... (suyuchen.warl wang, bang. bangg vulil@upenn.edd edu ge. ree.hhg@@@uwattrroo. averouz.taslakee sai sai.raafeswar com jiefuêust.hr
ABSTRACT
We introduce Visual Caption Restoration (VCR), novel vision-language task that challenges models to accurately restore partially obscured texts using pixel- level hints within images through complex reasoning. This task stems from the observation that text embedded images intrinsically differs from common visual elements and text due to the need to align the modalities of vision, text, and text embedded in images. While many works incorporate text into image for visual question answering, they mostly rely on OCR or masked language modeling, re- ducing task to text-based processing. However, text-based processing becomes ineffective in VCR accurate text restoration depends on the combined infor- mation from provided images, context, and subtle cucs from the tiny, exposed of masked texts. We develop pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, construct VCR-WIKI for VCR using Wikipedia images with captions, including 2.11M English and 346K Chinese training entities, plus 5K validation and 5K test entities in both languages, each in easy and hard configurations. We also make hidden test set VCR-HIDDEN to avoid potential fettifiting on VCR-WIKI Our results reveal that current vision language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-WIR and the data construction code to facilitate future research.
INTRODUCTION
Recent advances in large language models, such ChatGPT (OpenAl et al.] 2023) and Llama (Tourron et al.) (202)), ) have spurred significant interest and progress in the field of vision- language models ( (VLMs). With models like GPT-4V (OpenAl
ala..2023) 2023) and LLLvV (Liu al. 2023a 2024a 2023b) blend- ing textual and visual information, the intersection of computer vision and natural language processing has become vibrant research frontier. These integrated models aim to leverage the potential of vision and language modalities to understand and interpret multimedia content more effectively.
Machine rng whore Eom
glabe are by new Even if ass large as a slovelap, its m more afficient and cheaper large scale experimentss
Amidst this evolving landscape, we introduce VCR, a novel vision-language task designed to challenge existing models uniquely. VCR challenges these models to restore obscured texts within images, which demands intricate synthesis of text,
Figure 1: An example of the VCR task.
"qqual contribution.

---

Published as a conference paper at ICLR 2025
vision, and text embedded in the image. The VCR task is grounded in two key insights: (1) text embedded within images, with its characteristics different from common visual elements, represents distinct modality that requires careful alignment of vision, textual data, and the structure of written texts, and (2) neuroscience findings that suggest that humans are proficient in recognizing partially oc- cluded objects through sophisticated visual and cognitive processes (Thinés et al. 2013 Pessoa et al.) 1998; van Lier & Gerbino 2015 Fyall et al.) 2017, [i et al., (023a). By leveraging these insights, VCR seeks to explore how well vision-lange language models can handle texts embedded within images, aligning visual elements and natural language to mimic human-like multimodal understanding and recognition.
The Visual Question Answering (VQA) task (Antol et al. 2015 Wang et al. 2018 Mishra et al. 2019b Singh et al., 2019) has been popular benchmark in assessing how well models align and interpret visual and linguistic information. Traditional VQA mainly addresses visible elements, overlooking the nuanced relationship between embedded text and image context. This highlights the limitations of current models in handling integrated visual-textual data, especially when text is obscured or altered.
To address these limitations, our VCR task builds the premise that effective text restoration from images requires an integrated understanding beyond the capabilities of current VQA benchmarks. For example, in extreme cases, models rely existing Optical Character Recognition (OCR) system to extract text from documents (Singh et al. 2019 Borisyuk et al.] (018). The extracted text is then used context for generating answers without true semantic alignment between the text and the visual elements of the document. This approach, while effective in simple scenarios, falls short in more complex settings where text is intricately woven into the visual narrative of the image.
To develop the VCR task, in this work, we introduce a pipeline for generating synthetic images that allows for manipulation of the visibility of the textual components of the image. This not only enhances the challenge posed by the task but also provides a scalable way to adjust task difficulty. The resulting dataset, VCR- WIII, comprises 2. IM English data and 346K Chinese data sourced from Wikipedia, featuring captions in both languages across 'easy' and 'hard' difficulty levels. Our evaluations indicate that existing vision-lange vision-language models significantly underperform compared to human benchmarks, underscoring the need for novel model architectures and training paradigms specifically geared towards this complex intermodal alignment. We also constructed a hidden test set for the VCR task (VCR-HIDDEN) to avoid potential over-fitting VCR-WIKI
We release VCR- WCKI and its construction code to stimulate further research of developing of models that can more adeptly navigate the nuanced landscape of the restoration of text embedded in images to bridge the gap between human and machine perception. The code and datasets are available at GitHub and Hugging Face.
Contributions
The main contributions of this paper are:
C1 Introduce the VCR task challenge VLMs to restore occluded texts in images.
C2 Develop a pipeline for generating synthetic images with embedded text that allows for adjusting the visibility of such text, thus providing rich testing environment for VCR.
C3 Create and release VCR-WIKI, dataset with multilingual captions and construct the hidden test set HID-HIDDEN, designed to benchmark VLMs on text restoration tasks.
C4 Conduct empirical evaluations that show significant gaps between current models and human performance the VCR task. This highlights the effectiveness of VCR for assessing advancements in VLMs and underscores the necessity for innovative model architectures and training techniques. New models will be actively added to our Github leaderboard.
2 VCR TASK DESCRIPTION
In this section, we compare the VCR task with other existing tasks and answer the following questions:
Q1 What is the difference between VCR and other visual reconstruction tasks?
Q2 Why should care about VCR?
2

---

Published as a conference paper at ICLR 2025
For better clarity, we define text embedded in image (TEI) text incorporated within the image, visual image (VI) as the non non-textual portion of the image, and string text (ST) as the separate textual element associated with the image (typically the question prompt). A VCR task element can thus be expressed as (ST: (V1, TTT), where ST is a string while VT and TTT are presented in image form. We adopt this notation facilitate explanation, not to imply physical separation of V7 and TE' in the image. Please refer to Figure for an illustration of VI, TEI, and ST.
A1 VCR relates to both VQA and OCR tasks. VQA takes images and questions as input generate free-form responses with non-unique ground-truth, creating evaluation challenges regarding non- unique answers. In contrast to VQA, OCR is task where the ground-truth responses are unique: OCR takes as input complete characters in image form and outputs string representing the characters in the image, without considering the image context. VCR bridges these tasks by reconstructing unique text while considering visual context. can fill blanks easily, but models with only OCR capabilities cannot recover covered text without context, as pixel-level character hints no longer yield unique solutions.
(shows a hard-mode VCR task where humans
A2 The proposed VCR task is significant in two aspects.
First, it connects to fundamental neuroscience findings on human cognitive abilities to recognize partially occluded objects (Fyall et al., 2017, Li et al., 2023a). While existing models struggle with occluded information, humans excel by combining low-level visual processing with high-level cognitive functions in the prefrontal cortex. VCR serves as a critical probe distinguishing between low-level recognition and high-level reasoning cognition- distinction essential for advancing AI systems toward human-like perception capabilities.
Second, VCR presents unique chal- lenge substantially different from ex- isting benchmarks by specifically tar-
1. image very likeyysbeut menual plane.
geting text-image alignment capabili- ties. Unlike traditional VQA or oc-
AVION HANRIOT
2. visible pizel leval hist ① shows that is a worrdd ending "It". Given the previcus word is "rihttrr the frrst blank likaly "aircritt"
cluded object restoration tasks that test general visual reasoning, VCR creates a specialized evaluation frame- work that tests model's ability to maintain semantic consistency across multiple modalities simultancously. It requires deep integration of visual content (V with partially visible tex- tual elements (TE,), necessitates in- ference capabilities that go well be- yond pattern recognition toward genuine comprehension, and demands contextual reasoning that mirrors human cognitive processes when faced with incomplete information.
3. The tite of the manual mentloned "HANRIOT" which is French manufacturer.
4. 2 start from "F" combined
5. itiv likely "Francs". need *ad]"" "between "Frence" and "Woric Warl", Glven ( ends with
"s" and start with "d" We can infer it "during".
6. 2 stari from "p" and and with 1d" norch to
The Hanriot HD 2 was two-seat fighter
be vero. Connecting with itls "produced".
7. 3 as word star ting from "I" and only has evident that "in".
World War I.
Figure 2: How humans would possibly solve a VCR task.
What makes VCR particularly valuable benchmark its precision in targeting the specific frontier of aasion-languaa integration. By occluding text rather than objects, VCR creates a controlled environment where success requires sophisticated cross-modal reasoning rather than mere recognition or memorization. The ability to adjust difficulty through varying occlusion levels provides researchers with a finely calibrated instrument to measure incremental progress in model capabilities.
For practitioners developing next- next-generation multimodal systems, VCR offers several distinct ad- vantages: (1) it provides a reliable measure of text-visual alignment capabilities currently lacking in standard benchmarks; (2) it simulates real-world scenarios where text is partially visible, damaged, or obscured; and (3) it offers clear evaluation metrics with unique ground-truth answers, unlike subjective benchmarks that suffer from evaluation ambiguity. Figure demoonsratts how humans solve a hard-mode VCR task, highlighting the cognitive processes that advanced AI systems should aim to replicate.
DATASET CREATION
The VCR task requires aligning visual images (VI) with text embedded in images (TEI). Therefore, the dataset creation process relies set of highly correlated image-text pairs. We utilize the primary
3

---

Published as a conference paper at ICLR 2025
Wikijenlaa Imago Captions
Step Text Processing
Step Create TE1
Sitng D71
Republic Airport a regional sirpor in East mingdal,, located one east
(Peeuaiicc Aiment la regional) (Arport regionall alrport
Airport rogional East
irage? Phoss mustre Pe
vetical oaliding As Aeedkna"
Farmingdalo,
BROOE R. Prc.ittc Alr.
sirpori
localed cont
Farmingdalo villgoo
Exooiingta dlagg liil.
spaCy
Language Safety
Step 1: Data Filtering
Origical
wwg.hle segrana
Eas (Jew elocured))
all
Republic Airport a Farmir gaaai located
Republic Airport
Hiihh-Qaalis ImagoCCption Paire
Panctuataora
Digita
Farmingainn dutt -o
Republcc sport
Porsin, orginiotionn lacatin, date, time...
graligaale villgne kmits.
York. localed onip
Farmingcale
Filter iestaness withoot muskedin-grime
inggramte
Hard (mare obocuned))
Text endeeddedddii Image
Nppentic
bellww 50%s
Figure 3: Illustration of the dataset creation pipeline for VCR-WIKI, visual image (VT), text embedded in image (TEI) and string text (ST") in an example of the English Hard configuration of VCR-WIKI. The solid line-enclosed contents (V I and 7 T) I) are part of the image, whereas the dotted line-enclosed content (ST) is given separately from the image.
images and their corresponding captions from Wikipedia as the data source to create VCR-WIKI, Wikipedia-based VCR dataset. The pipeline for creating VCR-WIKI is shown in Figure 3 Before constructing the dataset, we first filter out instances with sensitive content, including NSFW and crime-related terms, to mitigate Al risk and biases
The VCR- VCR-WIKI dataset is formatted as a VQA task, where each instance includes an image, a question, and a ground-truth answer. The images are synthesized from text-image pairs by stacking the image (V 7) with its corresponding text description ETT vertically, mimicking the format of a captioned image. This stacked image is referred to stacked V I +T'E image. Each VI +TEI image is resized to a width of 300 pixels. To avoid excessive image height, we truncate TEI to a maximum of five lines. We filter the dataset to exclude instances with VT +TEI images exceeding 900 pixels in height to avoid drastic resolution changes during data pre-processing.
We use spaCy to randomly select 5 5-grams in the caption for masking. To ensure the restoration process is doable by a human without too much domain knowledge, the 5-grams do not contain numbers, person names, religious labeled by spaCy. The total masked token does not exceed 50% of the tokens in the caption. We pick 5-grams for masking it balances linguistic complexity and task feasibility, capturing meaningful grammatical structures while avoiding dataset reduction or overly simplified tasks observed with longer or shorter spans. We exclude instances that do not have any maskable 5-grams. The selected 5- grams are partially obscured by white rectangle that reveals only the upper and lower parts of the text, with the proportion of coverage varying according to task difficulty. Furthermore, to assess the impact of VI on model performance, we create an ablation for each image, maintaining the resolution of the VI +I image, but retaining only the TEI part in the center of the image.
political groups, facilities, organizations, locations, dates, and times
The VCR task involves predefined question that prompts the model to produce the obscured text in the image. The ground-truth answer corresponds to the caption displayed in the uncovered portion of the stacked image. Due to the extensive availability of VLMs and significant user base in both English and Chinese, we have chosen to develop the dataset in these two languages. For each language, we meticulously select the height of the masking rectangle to create two task variants: (1) an easy version, where the task is easy for native speakers but open-source OCR models almost always fail, and (2) hard version, where the revealed part consists of only one to two pixels for the majority of letters or characters, yet the restoration task remains feasible for native speakers.
To avoid test data leakage, we create a hidden test (VCR-HIDDES H-DDEN) for the VCR task which will not be publicized. While it follows our dataset's ' general construction principles regarding masked span length and the span selection criteria, differs in five key aspects: (1) Image-text pairs no longer come from the Wikipedia: (2) TEI is randomly positioned either above or below VI; (3) Masked regions are randomly selected from both "easy" and "hard" settings; (4) Multiple popular fonts are used randomly for the TET component; and (5) The VT ET image width varies uniformly between 600 1000 pixels. We include results on VCR-HIDDES in Table [ and Appendix
Datasource:
t_base
4

---

Published as a conference paper at ICLR 2025
3.1 DATASET FORMAT AND STATISTICS
The VCR-WIKI dataset comprises four configurations: English Easy, English Hard, Chinese Easy and Chinese Hard. Each configuration be further divided into training, validation, and test splits. The validation and test splits contain 5,000 entities each. The training set for English configurations and Chinese configurations contains 2,095,733 and 336,448 instances, respectively, which can be used for model continuous pretraining. To avoid test data leakage. we also newly include hidden test sets (VCR-HIDDEN) for both languages as described Section 3] which contains 100 entities for each language. We include detailed statistics of the dataset in Table in Appendix A
4 EXPERIMENTS
In this section, we report the experimental results of existing state-of-the- ofrtth-art vision-languaan models on both VCR-WIKI and the VCR-HIDDEN hidden test.
4.1 MODELS
Closed-source and Open-source Models. In this paper, report results for several state-of-the-e closed-source and open-source models from the OpenVLM Leaderboard" as well as selected state- of-the-art models of February 2025 for VCR-WIKI. For the VCR-HIDDEN hidden test, report results of state-of-the-art closed-source and open-source models available as of February 2025. See Appendix B for complete model specifications. We commit to evaluating emerging state-of-the-ar VLMs to reflect cutting-edge advancements. Results for later models will be actively updated in the leaderboard hosted on htttps:: cww wwooottiaaye 2 /CC
Fine-tuned Models. To test whether VLMs can learn to conduct VCR via fine-tuning, we select three models from the open-sourced models: Cog -2--222-m--3--11999111 and Diven2-VL-7B-Instee
MiniCPM-Llama3-2255 and fine-tune them subset of VCR's training set.
More specifically, we fine-tune CogVLMM2-Llamal 19B-Chh, MiniCPM-Llama3- VL-7B-Instruct in the English Hard configuration, and - 1-B---hhi---------------------- Chinese-Chate MiniCPM-Llama3- V2.5, and Qwen2-VL-785
and Qwen2-
on the Chinese Hard configuration. The models finetuned using LoRA (Hu et al.) (022) with 8 and 32. We adopt the schedule-free Adam W optimizer (Defazio et al. 2024) with a learning rate 2e-4. The effective batch size is 64. Each model is trained the first 16,000 examples of the training for epoch. All fine-tuning experiments are performed on a single node with 4 NVIDIA L40S 48G GPUs.
4.2 METRICS
We measure the quality of the model's restoration each masked n-gram (where n our setting, as specified in Section 3.. Due to the variability of different models' outputs, for each masked n-gram m Vr. where V. is the vocabulary of the evaluation tokenizer we extract the most similar n-gram i 4 with the least edit distance in the model's generation.
We report the two metrics below in our experiment section to measure the restoration quality: Exact Match (EM) EM (m, ri) = 1(m m). which measures whether the restored n-gram vi totally matches the ground-trute truth m; and Jaccard Index (J) [(m))sSm)] similarity of m and m as bag-of-words.
(8m))S(aa) which measures the
4.3 RELATIONSHIP TO OTHER BENCHMARKS.
We evaluated 38 Vision- LanguuLe Models (VLMs) across 23 different benchmarks, using the VLM performance scores as features of each benchmark to compute correlation matrix. Based on this
We selected the highest- open-source models with fewer than 40 billion parameters from https:/ httes://huggee /huggi inn acce. h/tps////s.eeeeeeeeceaaaaaa///ap their later versions for VCR-WIKI.
of May 2024
"e utilize spaCy's wet.corr__eb_mss and zh_core_web_sm tokenizer for English and Chinese evalu-
ation, respectively.
5

---

Published as a conference paper at ICLR 2025
matrix, in Figure 4 we applied K-Means clustering and visualized the results in 2D by plotting the first two principal components derived from the correlation matrix rows for each benchmark. Additionally, provide a heatmap of the correlation ma-
Clustering Benchmarks
trix in Appendix E
Cluttr
ground
VCRzu, EASY and VCRzi, HARD were cluded from these processes due to the lim- ited availability of VLMs that support Chi- nese. According to Figure I VCREN, EASY shows a tentative similarity to ChartQA and TextVQA, as all three benchmarks evalu- ate the ability to extract and reason about text from natural images and documents. However, EN EASY does not exhibit sig- nificant similarity to the other benchmarks. Meanwhile, VCREN, HARD stands apart from all other benchmarks. We attribute this to the fact that VCREN. HARD emphasizes cap- tion recovery with minimal pixel-level in- formation, a skill not tested by any of the other benchmarks. Therefore, we assert that the VCR series benchmarks assess unique aspects of VLMs that are not covered by any of the other benchmarks in our evaluation.
Choter
0.25
COOOLNN Poudhg
PCA Component
u.00
-b:25
-2.50
-0.5
0.5
1.0
2/0
PCA Co Commonmntt
Figure 4: Projection of benchmarks onto the first two principal components derived from the correlation ma- trix of VLM performance scores. Each point represents a benchmark, and proximity indicates higher similarity based on model performance correlation.
4.4 EXPERIMENTAL RESULTS
For VCR-HIDDEN, Table I presents state-of-the-art VLMs' performance as of February 2025, with more comprehensive results for more models in Table (English) and Table 7 (Chinese). In addition, Table 2 shows exact match scores and Jaccard indices on VCR-WIKL Figure comparees our fine-tuned models against base models across all VCR settings. For faster benchmarking, Tables and in the Appendix provide results on smaller VCR-WIK WIKI test sets containing 100 or 500 samples. In this section, we analyze models' performance as of our paper submission date (October 2024) on the VCR task, highlighting key insights through comparative evaluations.
VCR Remains a Challenging Task for SOTA VLMs. Despite high- high-performing models like Qwen2-VL excelling in VCREN, EASY and VCREN, HARD settings, most recent models struggle signifi- cantly, especially under harder settings where metrics approach zero. This highlights not only the inherent difficulty of the VCR task but also that subpar performance on VCR-WIKI stems from lack of reasoning capabilities or sufficient text-image aligument rather than unfamiliarity with the underlying text, as many VLMs are pretrained on similar data. These results emphasize the need for advancements in VLM designs to achieve robust performance across all settings. Besides, to avoid over-fitting on the train set we released, we also include results from our hidden test set VCR- HIDDEN, which is not publically accessible. For most recent models, we observe a performance compared with VCR-WIKI. We will keep updating the both the public "CR-WIKE and private VCR- HIRDH leaderboards in our Github repository.
decline on
Enhanced OCR Capabilities Do Not Necessarily Translate to Improved VCR Performance. Our analysis reveals that models proficient in OCR, such as InterL.--CComposer) VL. and those excelling in image document understanding, like DocOwl 1.5 and Monkey, demonstrate subpar performance across most VCR settings. This discrepancy suggests that while these models can accurately recognize text within images, they lack the advanced reasoning capabilities required to effectively interpret and utilize this information within the context of the VCR task. This finding also highlights a fundamental distinction between OCR tasks and the more complex VCR task.
Language-Specife Performance: Need for Enhanced Multilingual Capabilities. A significant performance degradation is observed when models evaluated on Chinese configurations, despite assertions of basic Baglish-Chinese bilingual capabilities. This decline is particularly surprising given the logographic nature of Chinese characters, which theoretically offer higher recognizability

---

Published as a conference paper at ICLR 2025
Table 1: Performance of some SOTA-level VLMs (as of February 2025) on the VCR-HIDDEN in English and Chinese. We label the best result of each setting and metric with bold fonts. A superscript of * marks that the model was released after the initial public release of the VCR-WIKI dataset (June 10, 2024). Subscripts show bootstrapped standard deviation. Refer to Table 6 for the complete results English and Table i for the complete results in Chinese.
Larquage
Opealined
Modellaane
Madel
FRact word (5)7
Jurved Indes (S3
72. -721
V2+7251
135
Gieaai 2.0 Mtack'
45 135
42
0.50
03.28
5.23
Cloaa
-180
14:00:25
.10
Clod: Vidio"
16 55
35
.450
35
45 45
.3.44
Englin
730
33 53
33 10
3.50
31 17
45 15.5am
13.75
4.0
27
Qy4
0x0 Previnal
770
06 15
1.0
75 25
73 27
314
25.50
7.30
02.38
3.55
Cloaa
Grndei 26 TTack'
1/3%
1.7%
1.%
0.50
1.30
1336.m
825
15 17
2.25
0.59
1.35
Oogttamt Chinee
1981
1.50
13 15
2.9
Chioc
QuQ-thrrize"
35 35.50x15
23.50
-5.00
24.7
34 380
-10.13
Queed 5-V1
(M
3.50
02.46
48.95-09
392
compared to alphabetic scripts (Wu et al. 2024a Zhao et al. 2022). These results indicate a critical
need for targeted improvements in multilingual support to ensure consistent performance across different languages.
Model Size Does Not Guarantee Superior Performance. Comparative analysis between Llama- BB and Llama-3.2-90B models reveals that both exhibit similar performance levels the VCREN. BASY and VCREN. HARD settings. This observation suggests that merely increasing model size does not inherently enhance VCR performance. Instead, advancements in the cognitive abilities of models, achieved through improved training strategies, reasoning frameworks, or architectural innovations, are essential for meaningful performance gains in VCR tasks.
Model Resolution Is Not Directly Correlated with Performance Enhancement. InternLM- XComposer2-LL.44K despite its higher resolution, demonstrates significantly lower performance on the VCREN, EASY setting compared to its lower-resolution counterparts. This decline may result from more aggressive image partitioning strategies that disrupt the spatial continuity of text or from more intensive pixel or token compression techniques that lead to the loss of crucial local details. Both factors are critical for the accurate interpretation required in VCR tasks.
Inclusion of V Input Images Negatively Impacts Performance. The addition of V generally results in negative performance changes (Á 0), indicating that the image information not being effectively leveraged by the models. This negative impact may stem from the importance of key information locations, which could be compromised by image partitioning strategies that fail to preserve spatial relationships essential for accurate reasoning.
Model Design Influences Performance Gains from VCR-WIKI Finetuning. As shown in Figure 5) finetuning on the VCR-WIKI dataset yields varying performance improvements across different model designs. Specifically: 1) CogVLM2 demonstrates substantial performance enhancements across all four settings after finetuning, indicating that its overall design may be well-aligned with the image-text reasoning demands of VCR, though further empirical validation is needed to substantiate this hypothesis. 2) CMn5CPM-V2.5 shows only marginal performance increases from an already low baseline, particularly in the VCRzH, EASY and VCRzH, HARD settings. This limited improvement indicates potential design limitations that hinder its ability to effectively address the complexities of the VCR task. 3) Qwen2-VL-7B maintains relatively high performance both before and after finetuning, implying that the model is sufficiently pre-trained on relevant tasks to perform well on the VCR task without extensive additional training.
We hope that through controlled variables, the VCR-WIKI dataset delivers fully comparable results across languages, difficulty levels, image inclusion, and fine-tuning stages. Each comparison is intended guide specific and targeted improvements in VLM development.
7

---

Published as a conference paper at ICLR 2025
Table 2: Performance of vision language models on VCR-WIKI in English and Chinese, for easy and hard modes. We label the best result of each setting and metric with bold fonts, the best open-source model with underline, and the best open-source model released before VCR-WIkI's initial public release (June 10, 2024) with italic font, A superscript of * marks that the model was released after the initial public release of -CR-WIKI. Subscripts show bootstrapped standard deviation. For more latest models' results, please visit our GutHub repository
Langaage
Mods
cearce
Medd rumse
Medd
Claske Cpel
37.0
-15
77:57
Goead Dro
02233
02.55
-31.5
77.70
01.3%
-13.5
Cloval
01.5%
16.5611
-301
96
07.06um
-1.3
Divon VL. Mic
4.71
857
-.74
Condensnl
335
41.5g
-150
01.7
18
32
06.41
1150
06.74
BecDull Ohas
50
125000
1771
-15
721
-116
Faraed MCceppeee
377631
3.95
433
300
13.5
Lianu3
118
12.32
31.50
SAT
Eagia
41.%
Malkay
18
565
-254
(2
7255
0.00
61.3
96.N
-3.6
122%
Dwvan VIL
733
91.%
-334
333
Draad VL.
4.53
91.6
4:25
VL-VL
4079
200
Caak Oypo
27
112
15.5%
-1445
Geaad: Pro
2607
16.0
19
10.8
-772
Clowd
-334
-1.5%
-1.27
41.5
2001
15
25.54
.099
Candeiuu
343
27.3%
20,86
2.48
2136
20.5%
2.60
Haat
18
14.
0779
15.
-1.
1001
100.
0.79
NCCoppord
1531
435
10
600
171
21. 43
0220
933
14
120
33.44
100
Milaky
18
1048
14/72-0
400
Dvilc"
1.15
42.32
030
Dwiedd V1
11
-133
50
-330
Yo-V
338
100
1.0mm
-138
Elawe 11pe
0330
-11
115
13.0
1.00
134
11.00
4035
LITE44 Teots
E3kaxe
550
0.55
Drep Kart V2.
1B
03.1
-377
3236.1
13.
127001
1200
-110
Opee
30a
18 15
11.34
Minkey
16
10
341
21726
16.9
2134
3.11
Owaech
17
37.33
73.82u
72.12.0
3.20
3.56
Clawk Sor
133
03
Clooa
23.00
13
2176.0
22.9
2111
Unim VL. Alex
12561
10.0
12.24
Eoghht MMT
12220
1103
16
213
1000
fenerd ww..eeeeeeetd.... vg.
11860
400
Feneen wepporedtet.e.. VL-K
NCC beies
615
401
11.00
435
Lobert
1237614
15
4016
MEaCCIS V2L5
58
6000
DA1
1200
100
05
Dvac.
345
1
3152
191
117141
01200
106
636
723
-1205
21764
-3391
125
22.47
2.06
54.03
106
TAIMS
231
8

---

Published as a conference paper at ICLR 2025
Original
100
83.33
80
Eont Matce (46
60
18.7
14
1.84
Cupata sW
Cayalt AM
Connine 2
Copartt Lve
Figure 5: Exact Match performance on VCR before and after LoRA fine-tuning for selected models. The results demonstrate varying degrees of improvement across different tasks and models, highlight- ing the heterogeneous responses to fine-tuning.
4.5 HUMAN EVALUATION
We recruited 7 volunteers to perform human evaluation on subset of the samples from our datasets. Two out of the seven evaluators are native English speakers, while five are native Chinese speakers who are also fluent in English" All volunteers have earned postgraduate degrees, majoring one of the following fields: biology, statistics, computer science, and economics. The evaluations were conducted on a voluntary basis, and participants received no rewards.
We gave the volunteers the following instructions: (1) We asked the volunteers to focus on the puzzles. Each example in the hard collection may require 30 seconds to minutes of focused attention, and (2) we asked the volunteers to utilize the context rather than directly brute-force the puzzle.
Every sample solved by at least volunteers. In English, we release the exact match score in 2 splits: all errors counted (All), and only counting errors not related to dates and person names (Filtered).
The human evaluation results are shown in Table 3
Although current SOTA models suffer from the challenge, fluent speakers can easily achieve more than 90 percent accuracy across difficulties. Please refer to Table g to compare all models with human evaluation results using the same test cases.
Table 3: Human evaluation results the VCR task in terms of exact matches. N is the number of puzzles in each language.
VCREN, EASY (N 169)
VCREN, HARD (N 169)
VCRzH, EASY (N 188)
VCRzH, HARD (N 188)
Mean (%)
SD (%)
Mean (%)
SD (%)
Mean (%)
SD (%)
Mean (%)
SD (%)
96.65
0.34
91.12
1.18
98.58
0.31
91.84
0.81
Filtered
98.62
97.63
2.13
99.47
96.63
1.11
5 RELATED WORK
Visual Question Answering (VQA). Several datasets have been proposed for visual question answering VQA (Antol et al. 2015 Zhang al. 2016; Goyal et al. 2017 Mishra et al., (019b). FVQA (Wang et 2018) and OK- (Mrrioo et al. 2019) are datasets about knowledge- based visual question answering and contains questions that necessitate the usage of external knowledge resources. CLEVR (Johnson et al. 2017) is a synthetic VQA dataset that mainly focuses on visual reasoning abilities. Recognizing the need to develop VQA models that understand text, Text-VQA. (Singh et al. 2019 Biten et al. 2019 Mishra et al., [009a]] Wang et al., (020)) aims to read and reason about texts embedded within images the context of image-question answering. Several datasets (Singh et al./ 20119; Biten et al., 20199; Mishra et al. 2019a) have been developed for the Text-VQL
task, such as the TeX dataset (Singh al. 2019 and the ST-VQA dataset (Biten et al. 2019)
*The TOFFL scores of the non-native English-speaking participants range from 102/120 112/120.

---

Published as a conference paper at ICLR 2025
on natural images. the OCR-VQL dataset (Mishra et al. 2019b) on book or movie covers, the InfographicVQA (Mathew et al., 2022) dataset on infographics, and the Voc dataset (Mathew et al. 202T on document images.
Vision Language Model. Vision-language models are designed for tasks that involve understanding
and generating content from images and text (Sun et al. 2023 Liu et al. 2023b Laurençon et al.,
2023
2024a). For example, models have been developed to combine Llama3 with advanced vision-
language processing capabilities to handle complex multimodal tasks (Yu et al. 2024 Xu al.
2024 Hu et al. 2023; Yu et al., 2023 Wang et al. 2023b, Dong et al.) 2024a). Qwen- VL (Bai
ct al. 2023) enhances visual-linguistic representations for more accurate contextual interpretations,
while OpenGVLab AppnGVLab-Intem' VL-Chat (Chen et al. 2023; 2024c) merges the InternVL framework with
interactive chat capabilities. These studies typically employ a multimodal encoder (Radford et al.
2021, Zhai et al. 2023; Wu et al. 2022) to process multimodal data, which is then mapped to the same imput space of the language model. General-purposs models such as the GPT-4 series models (Ouyang et al.) 2022 OpenAl et al. 2023), the Claude series models (Anthropic 2024), the
Gemini series models (Team et al. 2024a) and the Reka series models (Team et al., 2024b) have also been adapted for sision-language tasks, demonstrating strong performance in multimodal tasks. Finally, DocLLM (Wang et al. 2023a) specializes in document understanding by integrating visual and textual data enhance the interpretation and generation of document-related content. These models collectively represent significant advancements in vision-language integration, contributing unique capabilities and enhancements to the understanding and generation of multimodal information.
Optical Character Recognition (OCR). OCR Nagy 2000) and its subproblems (Howe 2013 Smith 1995 Shafait et al. 2008 Frinken et al, 2011) have been well-studied in the literature in the constrained setting. However, classical OCR methods often cannot perform well images captured in the wild in an unconstrained setting. Many new methods have been developed for advancing scene- text recognition on camera-captured images (Bissacco al. 2013 Gupta al. 2016: Huang et al..
2014 Jaderberg et al. 2014) Wang et al.) 2012; Shi et al. 2017: Zhou et al. 2017 Lee & Osindero 2016 In addition to the detection and recognition of OCR tasks, visual question answering has emerged as an important downstream task in the OCR literature. With the development of Text-VQA, new methods for improving the reading abilities in VQA utilizing OCR have been proposed. For example, LoRRA (Singh et al. 2019) extends a VQA model Pythia (Jiang et al., (018)) with an OCR module to better handle Text- VQA tasks. TAP (Yang et al.) (22T)) incorporates scene texts that are generated from OCR engines during pretraining to further improve VQA capabilities.
6 CONCLUSION
In this work, we introduced the VCR task, novel rision-languae challenge aimed promoting the integration of visual and textual modalities, including text embedded in both natural language tokens and image formats and highly obscured text embedded in the image. We developed specialized pipeline to create a dataset tailored to this task, utilizing correlated image-text pairs. This task stands out from existing methods by requiring more profound integration of visual cues and partially obscured text, highlighting its uniqueness and importance in the field.
We conducted extensive evaluations of state-of-the-art vision-language models (VLMs) in both English and Chinese. The results demonstrated significant room for improvement, suggesting that current models have not yet fully exploited the capabilities necessary for VCR. We selected models representing both the highest and average performance tiers for additional fine-tuning with our dataset. Although fine-tuning exhibited potential for enhancing VCR capabilities, it did not consistently result in significant improvements, indicating the complexity and challenges of adapting models to this task.
By introducing the VCR task and its specialized dataset, we aim to advance research in vision- language interaction. The unique challenges of VCR seek to improve model development and training, extending the limits of multimodal AI. VCR provides a controllable testbed for fine-grained analysis of model behavior across languages, difficulty levels, image inclusion, and fine-tume mu-tttnnn stages. We invite the community to utilize our dataset and develop innovative strategics to boost the performance of vasion-l models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
OL. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01 ai, 2024.
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim. Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmaan, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Teodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang. Lijuan Wang. Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jilong Xue, Sonali Yadav, Fan Yang, Jianwei Yang. Yifan Yang. Ziyi Yang, Donghan Yu, Lu Yuan. Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv: 2404. 142/9, 2024a.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report. arXiv preprint arXiv: 24 2412.0890 08995, 2024b.
Anthropic. The claude model family: Opus, sonnet, haiku. 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv: 2308. 12966. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv: 2502.13923. 2025.
Alessandro Bissacco, Mark Cummins, Yuval Netzer, and Hartmut Neven. Photooer: in uncontrolled conditions. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
Reading text
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Mareal Rusinol, Ernest Valveny, C.V. Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), October 2019.
11

---

Published as a conference paper at ICLR 2025
Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar. Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 18, pp. 71-79, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355520. doi: 10.1145/3219819. 3219861. URL httph:tts: wttp///iiorr/oooooooe 10. 10011144/3211 981999991191 321 9861
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv preprint arXiv:
2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv: 142.3888 2023.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye. Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren. Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang. Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge. Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open- sourcs multimodal models with model, data, and test-time scaling. arXiv preprint arXiv: 1412.0527,, 2024b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv: 1602,, 2024c.
Aaron Defazio, Xingyu Yang, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, and Ashok Cutkosky. The road less scheduled, 2024.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tammay Gupta, Kuo-Hao Zeng. Jon Borchardt, Dirk Groeneveld, Jen Dumas, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv: 2409.17146, 2024.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li. Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao. Dahua Lin, and Ji- aqi Wang. Internlm-xcomm xcomposer2: Mastering free-form text- -ixt- composition and comprehension in vasion-language large model. arXiv preprint arXiv: 2401. 16420, 2024a.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang. Yining Li, Hang Yan, Yang Gao. Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim-xcos scomposer2-4khdd A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint MXXX::44440061222 2024b.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal. Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, and et al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
12

---

Published as a conference paper at ICLR 2025
Volkmar Frinken, Andreas Fischer, R Manmatha, and Horst Bunke. A novel word spotting method based on recurrent neural networks. IEEE transactions on pattern analysis and machine intelli- gence, 34(2):211-224, 2011.
Amber M Fyall, Yasmine El-Shamayleh, Hannah Choi, Eric Shea-Brown and Anitha Pasupathy. Dynamic representation of partially occluded objects in primate prefrontal and visual cortex. elife, 6:c25784, September 2017. ISSN 2050-084X. doi: 10.7554/cLifc.2578.
Team GLM, Aohan Zeng. Bin Xu, Bowen Wang, Chenhui Zhang. Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang, Jiale Cheng. Jiayi Gui, Jie Tang, Jing Zhang. Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang. Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang. Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong. Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from 1m0b to glm-4 all tools. arXiv preprint arXiv: 2406. 12793, 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
Nicholas R Howe. Document binarization with automatic parameter tuning. International journal on document analysis and recognition (ijdar), 16:247-258. 2013.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv: 2403. 12895, 2024a.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference Learning Representations, 2022. URLhttps: http://ppppeee.eeeeeeee www.:neeffooume
id=nzeVKeeFyf9
Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint 2308v:2308. 12038, 2023.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li. Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024b.
Weilin Huang, Yu Qiao, and Xiaoou Tang. Robust scene text detection with convolution neural network induced mser trees. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 497-511, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10593-2
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Deep features for text spotting. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 512-528, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10593-2
13

---

Published as a conference paper at ICLR 2025
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Proehl, Enoch Cheung. Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Fran- cis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambatista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Ope- nai ol system card. CoRR, abs/2412. 167200 2024. doi: 10445500ARXIV..412.600 /tt::://ooi...
URL
/tttps://all.oo//0..8888 wttpo//......////// arwiv. 2412 16720
Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0.1: the winning entry to the vqa challenge 2018, 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kicla, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interlcaved image-text documents, 2023.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understand- ing vision languaee models: insights and future directions. arXiv preprint arXiv: 2408. 12637, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024b.
Chen-Ye Yu Lee and Simon Osindero. Recursive recurrent nets with attention modeling for oer in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 2016.
Bao Li, Chi Zhang, Long Cao, Panpan Chen, Tianyuan Liu, Hui Gao, Linyuan Wang, Bin Yan, and Li Tong. Brain Functional Representation of Highly Occluded Object Recognition. Brain Sciences, 13(10):1387, October 2023a. ISSN 2076-3425. doi: 10.3390/rrainscil 3101387.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv: 2311.06607, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances Curran Associates, Inc., in Neural Information Processing Systems, volume 36, pp. http:/ppapr_ffeeeeeeeee wwwp:fffeeesppppeeeeeee pttps//2ppr//22 /023/ 2023b. URL ht httpt:ttt///tttt http:::ss//pprooeeeennnggss neu neurips. wwtppprr.cpppppppppppmpp.m
File/ /ttlee/dccf2/eaaa222223332
www.peer-oooffeeece.po pdff
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Lexva-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https
7/1 lava-vl . tht....... httpbl/oo//oooo//o///0/00/
14

---

Published as a conference paper at ICLR 2025
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmon- key: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv: 2403.04473, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseck-vl: Towards real-world vision-language understanding, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Minesh Mathew, Dimosthenis Karatzas, and Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2200-2209, January 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision ( (WCV)., pp. 1697-1706, January 2022.
MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang. Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan. Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng. Lin Zheng, Linbo Chai, Long Xing. Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang. Qidi Xu, Qiexiang Wang. Qin Wang, Qiuhui Li, Ruitao Leng. Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang. Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang. Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, and Zijia Wu. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv: 2501.08313, 2025.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Oer-vqa: Visual question answering by reading text in images. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 947-952, 2019a. doi: 2,.1109//DARR2211.0001
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 International Conference Document Analysis and Recognition (ICDAR), pp. 947-952, 2019b. doi: 10.1099/CDDAR.2119.00..1
AI Mistral. Announcing pixtral 12B, September 2024.
G. Nagy. Twenty years of document image analysis in pami. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(01):38 12.3388--1111 jan 2000. ISSN 1939-3539. doi: 10.1109/34.82482
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, et al. Gpt-4 technical report. arXiv preprint arXiv: 2303.08774, 2023.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Luiz Pessoa, Evan Thompson, and Alva Noë. Finding out about filling-in: A guide perceptual completion for visual science and the philosophy of perception. Behavioral and Brain Sciences, 21 (6):723-748, 1998. doi: 00.1017/5110555599800
15

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021.
Faisal Shafait, Daniel Keysers, and Thomas Breuel. Performance evaluation and benchmarking of six- page segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(6):941-954 2008
Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- based sequence recognition and its application scene text recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 19(11)22298 2304, 2017. doi: 10. 10.1199//AMM.221.666666
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Ray Smith. simple and efficient skew detection algorithm via text row accumulation. In Proceedings of 3rd International Conference on Document Analysis and Recognition, volume 2, pp. 1145-1 148. IEFE, 1995.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang. Yuhang Zang. Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. arXiv preprint arXiv: 2312.03818, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv: 05530, 2024a.
Qwen Team. Qvq: To see the world with wisdom, December 2024. URL sttpss//qwents github. wttpiii///////ooo qvg- 72b mobb-preeeww//
Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d' Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Fugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang. Yi Tay, Yuqi Wang. Zhongkai Zhu, and Zhihui Xie. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv: 2404.12387, 2024b.
G. Thinés, A. Costall, and G. Butterworth. Michotte's Experimental Phenomenology of Perception. Routledge Library Editions: Phenomenology. Taylor & Francis, 2013. ISBN 9781 134506897. URL httts:://poook:k
google. com books wwwookssiinnnnRaaee. AbREAQAAQBAJ
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya lyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: 1: Fully Open. Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406. 166.666000,66600, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn. Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tunce chat models. arXiv preprint arXiv: 0928.0988. 2023.
16

---

Published as a conference paper at ICLR 2025
Rob van Lier and Walter Gerbino. Perceptual completions. In The Oxford Handbook of Per- ceptual Organization. Oxford University Press, 08 2015. ISBN 9780199686858. doi: 10. 0099/ooffort///77111166666663333
9780119686858 013.0.4..0 040
Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. Doclim: A layout-aware generative language model for multimodal document understanding. arXiv preprint arXiv: 2401.00908, 2023a.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel. Fvqa: Fact-based visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (10):2413 2018. doi: 10. 1099/TPAM..011.2275.5
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing lsiounlanguage model's perception of the world at any resolution. arXiv preprint arXiv: 2409.12192 2024.
Tao Wang, David J. Wu, Adam Coates, and Andrew Y. Ng. End-to-end text recognition with convolutional neural networks. In Proceedings of the 21st International Conference Pattern Recognition (ICPR20/2), pp. 3304-3308, 2012.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogylm: Visual expert for pretrained language models, 2023b.
Xinyu Wang. Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and liangwei Wang. On the general value of evidence, and bilingual scene-text visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Shilian Wu, Yongrui Li, and Zengfu Wang. Chinese text recognition enhanced by glyph and character semantic information. International Journal on Document Analysis and Recognition (TD):4) March 2024a. ISSN 1433-2825, doi: 10. 10.10007/1100322422--044444 URL
1 www.:100011/1.. 023 00444-9
Yusong Wu, K. Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kirkpatrick, and S. Dubnov. Large- scale contrastive language-audio pretraining with feature fusion and seyword-to-captinn augmen- tation. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2022. doi: 10.1 1A110//CASSSP4935..22222 209.94999957222...00999999999 10095969.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu. Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-vID Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv: 24 10302, 2024b.
Ruyi Xu, Yuan Yao. Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403. 2024.
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8751-8761, June 2021.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. arXiv preprint 2XXX::332.00044 2023.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Rlaif-v: Aligning milms through open- open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint a405 17220, 2024.
17

---

Published as a conference paper at ICLR 2025
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. IEEE International Conference on Computer Vision, 2023. doi: 10.1 109/
ICCV51070.2023001000
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang. Hang Yan, Conghui He. Xingcheng Zhang. Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm dneemlm-xcompose--. versatile large vision language model supporting cong-contextual input and output. arXiv preprint arXiv:2407.033320, 2024.
A
Peng Zhang, Yash Goyal, Douglas Summers-Stae Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv: 2302.00923, 2023.
Yuliang Zhao, Xinyue Zhang, Boya Fu, Zhikun Zhan, Hui Sun, Lianjiang Li, and Guanglie Zhang. Evaluation and recognition of handwritten chinese characters based on similarities. Applied Sciences, 12(17), 2022. ISSN 2076-3417. doi: 10.3390/app22118521. URL (ttptt/: /www
indpi. 0tmpi.ccom20700 2076 31777766417112277/88555
Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: An efficient and accurate scene text detector. Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), July 2017.
18

---

Published as a conference paper at ICLR 2025
A STATISTICS OF VCR-WIKI AND VCR-HIDDEN
This section provides comprchensive statistical analysis of both main dataset (VCR-wIKI) and our hidden test set (VCR-HIDDEN) Table 4 presents key metrics including split sizes, image dimensions, and text obscuration patterns across languages. Note that the English and Chinese configurations maintain consistent statistical properties between their respective Easy and Hard variants since the two difficulties keeps the exact same dataset splits and covered texts, and only differs in covered area.
Table 4: Basic statistics of VCR-WIKI and VCR-HIDDEN. Note that each language's s Easy and Hard configurations share the same statistics. We report the mean, standard deviation, and the 5th and 95th percentile (7).5 and 71.55) for the stacked image height and the number of obscured text spans of VCR-WIKI. Unit is in pixels. "Hidden Test" means the hidden test set CCR-HIDDEN).
VI +TE/ Image Height
Obscured Text Spans
Train
# Val
# Test
Hidden Test
Mean
SD
4.5
4.95
Mean
SD
%
1/.6
English Chinese
2095733 336448
5000 5000
5000 5000
100
375.52
106.01
253
564
1.62
0.63
1
3
100
360.44
102.76
239
562
2.06
0.94
1
4
B DETAILS ABOUT THE EVALUATED MODELS
This section provides comprehensive specifications of vision-language models (VL.Ms) evaluated in this study. We assess diverse range of models spanning both proprietary and open-source ecosystems, representing the state-of-the-art in VLM capabilities as of October 2024 for VCR-WIKI and February 2025 for VCR-HIDDEN Our evaluation includes models with varying parameter sizes, architectural designs, and training paradigms to ensure a thorough assessment of frontier visual- textual reasoning abilities. The models categorized below either closed-soures (proprietary) or open-source, with detailed specifications and links to the models provided in Table
Closed-source Models. We evaluate several most advanced proprietary models with either their official APIs or APIs provided by OpenRouter. The evaluated models include ol (o1-2024-12, 12- 17), GPT-4o
GPT-4 Turbo
wuuuu--o02-000000 04-09), GPT-4V (gpt-4-1 106-
vision-previewt (Jacch et al. 2024 Ouyang et al. 2022 OpenAl et al. 2023), Claude Opus
Claude 3.7 Sonnet (Anthropic 2024). Gemini 1.5 Flash (gemini-1.5-fat 1.5-ftash--h), Gemini Gemini 2 Flash Lite 1eemini-2.0-flash--t Gemini 2 Flash (gemini-2.0- flash) (Team et al. (2024a), Reka Core (reka-core-20240500) TTem et al. 2024b), and Qwen-VL-Max VL-Max (tested in May 2024)
(llude-33-puus222242222 Claude 3.5 Sonnet
(claud-3---somnee22255002
1.5 Pro 1eemini-1.5-pro-000
Bai et al. 2023).
Open-source Models. We evaluate open-source model families with the best performance on the VLM Leaderboard as of May 2024 and state-of-the-ant Chinese VLM models. The evaluated models include Cambrian-l (Togg et al. 2024), CogVLM2-199 (Hong et al. 2024).
DeepScck-VL-77-CCK (Lu et al.] 2024). DeepSeek- VL2 Series (Wu al. (222b)), Dwwowl-Oooo
Omni Huu et al., 2024a), GLM-4v (GLM et al., 2024). Idefics2-BB (Laurençon et al. 2024b).
Idefics3-8B 8B (Laurençon et al.) 2024a). Intern VL 1.5. Intern InternVL 2.5 (Chen et al.
2024c b). InternLM-XX LLM.---ooosse2-VVV VL-7B (Dong et al. (2224)). IntemLM-XCooppeeee (Zhang
et al. 2024), InternI.M-XL W-I.-M---ommmpm XCooposer2 VL.4KK (Dong et al. 20044).. Llama-3.2 (ubey et al. 2024).
MiniCPM- MiniCPM-V2.6 (Hu et al. (024b), MiniMax-ViL VL-00 (MiniMax et al.) 2025), Molmo
Series (Deitke et al. 2024]. Monkey (Liu et al. 2024bb Li et al., 2023b), Phi 3.5-vision (Abdin
et al. 2024a), Phi-4-multmodal (Abdin et 2024bb). Pixtral (M24), Qwen-VL-Chat (Bai et al. 2023), QVQ-previee
(Team] 2024). Qwen2- VL Series (Wang et al. 2024). Qwen2.5-VL Series Bai et al. 2025) and Yi-VL (01.Al et al. 2024). Out of these models, Cambrian- Idefics3
and Llama-3.2 are English-only models, and Cog LLM2-Llaaaa 111-3-999---hhh has its Chinese variant, Cog CogVL.M2-L.Jamas 19BB Chw.amma--------ii---CCCCCC Chinese--Cha..
19

---

Published as a conference paper at ICLR 2025
Table Model specifications
Model name
Model size
Open-sourced
Claude 3 Opus, 3.5 Sonnet, 3.7 Sonnet
Gemini 1.5 Flash, 1.5 Pro, Flash, 2.0 Flash Lite
GPT-4 Turbo, 4V. 4o, 40-mini, ol Owen-VL-Max
X
Reka Core
Cambrian-1 Series
8B, 13B, 34B
CL2VLM2 English, Chinese
19B
DeepSeek-V VI. Series
1.3B, 7B
DeepSeek-V1.2 Series
16B, 28B
DorOwl-1.5-0mmf
8B
GLM-4V 10
9B
Idefies
8B, 8B
IntemL Xww.-M-Xooomm--2-2- 4KHD 13
7B, 7B
IncmLLM-XCooposer2...
7B
IntemVL-V1.5
26B
IntemVVL-V2 Series
IB, 2B, 4B, 8B, 26B, 40B
InternVL-V2.5 Series
IB, 2B, 4B. 8B, 26B, 38B
Llama-3.2- Series
11B, 90B
MiniCPM-V2.5, V2.6
8B, 8B
MiniMax VL-01
456B
Molmo Serie
IB, 7B, 7B, 72B 7B
Monkey
Series
9B, 27B
Ovis2 Series
IB, 2B, 4B, 8B, 16B, 34B 4B
Phi-3.5-vision
Phi-4-multimodall
6B
Pixtral
12B
Qwen-
7B
Qwen2-VI. Series
29
2B, 7B, 72B
Qwen2.5-Vi. Series
3B, 7B, 72B
QVQ
72B
Yi-VL
6B, 34B
20

---

Published as a conference paper at ICLR 2025
C HIDDEN TEST RESULTS ON VCR-HIDDEN
Table 6: Performance of vision language models on VCR-HIDDEN in English. We label the best result for each setting and metric with bold fonts. A superscript of marks that the model was released after the initial public release of the VCR-WIKI dataset (June 10, 2024). Subscripts show bootstrapped standard deviation.
Open//lesed source
Model name
Model size
Exact match (5)
Jaccurd index (6)
V7 PI+TET
TEI
V/+TE
TB8
Cloude 3.5 Sonnet
75.50g
78.342.0
3.00
55.12).9
56.55ly
1.51
Cluck 3.7 Souact
74.00g
58.50g.3
15.50
81.53g.14
61.471.01
20.06
GPT-4o
41. 16.66..
45.50g,35
-4.00
05. 65.0P%u2
51.51g.m
-3.43
GPT-4o Mini el
00.500 00.500..4
60.000.29
-350
T1.022.94
$1.192.11
10.17
Closed
20. 91517
18.54 18.502
3.00
$1. $1.17
22.00
9.11
Genini 2.0 Flack
88. M0.55
42. 12.00
6.50
06. M6,25
61.16 61.16,,0
5.20
Genini 2.0 Flash Lite
50.10
32.00
7,00
51 92:
16.
6.63
Cemini 1.5 Flach
57 01
41.00..0
-7.00
74
35.50x14
-8.1
Cemini 1.5 Pro
57 56
47.00
-9.50
62.25
-8.75
Gmi 2 Vision
36,03
77.78
Camleia--I"
138
27. 500
56.
-20.00
61.3 g4.54
7..4
-0.00
Cambrian
39.50
57. 57.15
11.11
CogVLM2
198
51,00
2,00
64.97220
3.97
DecpSeck V1.2 Seall
5,00
6,06,
24.74
DeepSeck
23,50,,33
75..34
20
035.6
Neelicd
18.00,.79
IncmLMM XComposer22
9.50
7.50g 85
23.49p.14
InsemLM XComposer2
50
14.133.99
Intend 2..CCepaa22....
IsteVVL1.5
20.00
ImtmV1.2
000.s
13.0 24
4.00
7.04
ImemVL2
10.50u.1
11.50,
3,00
25,0 35.005..1
76
6.20
48
17.00g
15.50..51
1.50
$1. 66,
12.12
2.05
1mteVL22"
80
22 (00g.9
15. 22,54
6.50
30 64g.4
31
7.65
IsteV122"
26R
01'
16
1,50
1.75
InemVL22
IatnVV12
IsemVL255
70.00
74.01.70
IstemV1...5
74.00
ImemV1225
63.00
11.50
74.
9.91
ImcnVL1..
268
88. 00,000
$4.00
4,00
94.60,
D0.17
4.43
34B
70 20
78
1,00
99.72).49
87.00
2.65
Open
V1..5
78R
83.
79
4.00
01.47..9
3.87
Llama 32-Vision
50.
00:1 02.12
Llama-3.2.-0 3.-Visii
51.00
MinCCMM V2.5
15.000
26
MinICPM-V26"
VL
M0 52
23.
Melano
27.00%
66
MeiimR
0.00
5.8%v.
9.9%
-..12
Meimo
50
16.00g
6.50
40.05
34.52
5.51
Meimo O'
70
20.50
18
2.00
00.52:.1
11 (00g
-0.5
Ovis 1-11.6-40 G-GGnma2
9B
41.00 11.00
37. 20
3.50
54.075111
51. 10
3,58
Osis 6-Cermma2
27R
58 35.00s
42. 50
-4.90
52.45 02.45.49
$1.51
-2.96
Ovis2
56,00
74.45
6.17
06.13
20.50,
72. 72.13..v
72
17.00..5
10,00
93,
Ovis2'
34B
(00
60.500g.
1.50
73. 20.20
71. 14
1.97
SiiiinS-
481
12.00
15.00
-3.00
25 5...
-2.93
Pti bautimoda"
68
6,00y
16,0
10,00
17.47:.55
20.11
11.69
Piatna
12B
27 500.0
19.00 19.00
$.50
00.05
4,45
Previee
62.5
3.50
3.19
Qwen-Vi VI.
15.50s
12.50
3.00
5.47
Owen2 VL"
96.25.344
Owenl-VL
$0.00
86.502.44
00.62,47
Owen2.5
65.00,
66,00,
73.02
-3.10
Owen2.5 VL."
78
58.00
74.00um
16.00
65, 75..5
53.50
18.15
V.V..
72B
86.50
70.50
7.00
9d. 00.25
86.43um
5.86
21

---

Published as a conference paper at ICLR 2025
Table Performance of vision language models on VCR-HIDDEN in Chinese. We label the best result for each setting and metric with bold fonts. A superscript of marks that the model was released after the initial public release of the VCR-WIKI dataset (June 10, 2024). Subscripts show
bootstrapped standard deviation. Model
Open/clooed
Exact match (36)
Jaccard (16)
source
Sonnet'
3.000
0.50g.5
9.98.4
0.92u
0.500.50
3,78
2:222:05
0.00ksss
0.00uus
0.00A.so
6.00ua
Closdd
Gemini Flach"
1000.73
0.00
12.36
11.77.u
0.59
Gemint 2.0 Flash Lite
0.5
0.50
0.00
7.47
6.97uu
0.50
Gemini 1.5 Flast
0.5
0.50
0.00
5.5
9.90uu
1.39
Gemini 1.5 Pro
10
0.00
12. 12.12
10.50
1.21
Grok 2 Visios"
0.00
0.00p
0.00
6.276
5.8%
0.94
Canteeian-
138
0.
0.00uus
0.00
4.0
6.8%ur
-2.76
Cambrian-1
348
0.00kes
0.00
0.00
1,215
4.00n
-366
C2- M2-CCinnnn
196
3.00
1.50
1.50
16. 16.0%
13. 15.15,0
2.90
DeepSeck VL2 Small
168
0u
0.00g
0.00
1.3
2.5
1.26
DespSock-VL.2
258
0.00
0.00
0.00
6.82
7.0%
-0.26
GLM-4.
1.50
2.00g
-0.50
14.07:.94
6.75 6.75
4.85
Mlchie2
0.00y
0.00
1.5
2.85
-0.83
3dedice"
IntenM XComposer2 VL
I XCompeser2 VL-4K
0.00
8.07
8.07e.3
Inend Mc-XCoopsc22....
10.53ua2
2
10.00.44
2.0
0.000.4
11:29h.0
InteVVL2"
1.00aa
10:11aa5
Inte/1.2"
1.50
0.50
10.27
1.28
ImemVL2'
0.5
0,50
0.00
10.92..3
9.13
1.83
Imte
40B
2.00kss
0.00
2,00
13.121.33
11.30
1.82
1menVL22
76B
0.500.
1,00u
-0.50
12.05 12,05
11.88
0.78
1m.mm11..5
10.00
10.00
50,
11.21
ImemVL255
17.44
IacnV11..5
InseVV125
$1.00,
10,50
54.71.s1
Intem/11..55
12.50
7.00g.7
10..1
InteVV1..5
21 (00
9.50
25,
11,43
Llama-3 3.2-Vision
11B
0.00
0.00
0.00
1.8%
-2.41
Llama 3.2 Vision
90B
0.100u00
0,00
11.356a1
9.77 9.77671
1.61
MiniCPM V2.5
89
0,50
0.50
7.62
5.000.75
2.61
MisCPMM-V26
88
1.50uss
L.
0.00
9.20,
9.97
-0.77
MiniMa--VL-Or
4568
2.0
2.72
Meimo
5.65u.54
Meimol
0.00uss
0.00ua
3.02u.5
Melmo-O"
0.00
0.00
4.2 44
Ovis Genmaz"
1.000.22
0,50
0.50
6,50u.5
8.05
2.45
Owis 6-Ceenmna2
27B
1.50a
0.50
10.57
9.5
0.86
Ovisz
10
19,00s
15,00
4,00
41.7%
35 21.
6.59
Ovis2
28
24,00gas
15.00
9,00
64.65g.33
36.57
5.27
Ovis2'
5.00,
4.50
1.50
24.7 70,7,
20.17
4.23
Ovis2"
2,5h
12.79,11
12.31
2000,00
1.0
12,39
29.
0.00kee:
0.00u
0.00A.s
0.11aus
Mutomodal"
0.00ka
0.00u
0.00
0.06
0.35
0.30
Piatnal
12B
0.00pu
0.00
5.47
8.67u445
-0.15
QuQ--raviee
726
18 50
23.50m
-5.00
24.7
34.55
-10.13
Qwen-Vi
78
0.00
0.00
0.00
0.00cs
0.0
0.00
Owen2 VL.
20
30,00,11
37. 37.30
7.50
47.129:73
55, 70
5.04
Owen2.V1
42.50g.2
46.50,.s
-3.00
60.56y..0
08.00 08.09,.42
-7.54
Qwen2-VI. VL."
$8.00,4
11.50yas
-3.50
56.81 58.812.5
61:00
-3.12
Owee2.5
8.50kss
24.24.77
Owcnd. S-VE."
17.00g
21.
$2.6%,9
42.54
Q.L.
31,00,
28.00
52.67
46.53,72
22

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EVALUATION RESULTS ON FIRST 100 AND 500 TEST CASES Table Results of various open- source and closed-source vision language models on the VCR task using the first 100 test cases. FT = fine-tuned on 16,000 samples from the VCR-WIKI training set. We label the best result of each setting and metric with bold fonts, and the best open-souce model with underline. Subscripts are standard deviations obtained from Bootstrap. For more latest models' results, please visit our GitHub repository
Largep
Broet vork USV
Jerwel Indes UNIT
T12
Dledio 10
10.6%
01: 11:1
11.05
Gredk Phe
T1.33
-15.0
10.00
04.21130
-1133
Cieek
55.8kmm
-479
14:77
18:03
Onne
08.11
332
Erg 141
190
175
Ray
31
-101
18
35
123
3009
Djcs
Idtwel 14 30 XT-4X
18
1.50
0155
6.6
40
05.850
41.0
198
14
Desd
241
-1.9
4030
m
-35
16
730
-1.55
730
113531
$135
216
00
3.54
Clidle Seame
457
3.50
137 m
5415
3-1
6-11
12:44
1623
-17
2.41
X11
18
-110
Dhal Dew
04
18
030
1000
4030
Upes
Hamel sonperett
17h
Iawalt: 3.2
38
3330000
206
6.30
23.00m
Mana 196-13
18
Mast 100.23692235
16
11hr
1916
Poats
130
6080
0.50
196
18
7574
11
127
636
710
4.0
0.29
1Phur
-1.1
Clidkc Ogga
033
003
1135
22
0130
0151
1537717
6:16
Clowl
Tedl:
0.53
3.56
Hotts Car
1036
033
Dog (1LLMM33000 M1
118
233
Doe V1.3
700
211
1.00
18
1/8
035
Hemel 14 irngeret
0ym
Heivett..2 -12
790
1257
af PSA
1186
18
1850
12
Ovid
300
16
6.16
Gread
730
78.7
-706
66.1
16.8
-336
Qecr25-VL
730
27.15
42305uu
-179
0653
-2.7
Chaddo Stmeet
ncdiv:
0.5
1:1mm
335m
6.6
G754g
222
1/6
11f03
-.5
Doghi
Dog'1
1191
0355
V1.
-320
210
-325
Idlwel 24 Mirggoow XT
16
1100
110
Dyea
filwo' V11
39
00206
1951
Maad V54
18
130m
-245
Monkey
18
0350
1
-8.7
Owne
130
12.0p
-3811
5158
Qpeap L.
4228
18
530
2.15
VL.
340
03
m
355
3.55
23

---

Published as a conference paper at ICLR 2025
Table 9: Results of various open-source and closed-source vision language models on the VCR task using the first 500 test cases. FT = fine-tuned on 16,000 samples from the VCR-WIKI training set. We label the best result of each setting and metric with bold fonts, and the best open-souce model with underline. Subscripts are standard deviations obtained from Bootstrap. For more latest models' results, please visit our GitHub repository
Langage
Medd rumas
Model
Boac Baac (N)
2B7
V1+
7E7
Claak Suredt
05.5%
723
-3344
78.35
45.00 15.0..1
-330
L1714 Tonto
41.36
-33
86.3%
1115
-3.0
(PF-4)
2286.
5417
01.30
11:33
Boks Cree
05 13
26.1.0
1545
3033
6.23
Carbettr
1G
26.9
332
Dag'1
079
03
17.
V8.5
18
4006
Barand Nccutpeart SVV
18
42-45.13
20.50
1338
103
01.36
10.00
M. 33
90
71.22.1
4.30
BUSK
4-5
$5.36015
55.33
Miskky
TELTT
343
-129
Piozaal
11.00-1 13
R36
11376
113
Yo-VL
345
135500
128
200
37
113
17.5%
J0185
MARI
0230
CPPT-
62
622
318%
227543
01.4
SUNCO
1118.15 15
141
3G
25.5%
3.06
2.5
Englt M4
140
$2.3%
19.0
Ereey fook VE.
$1.0
17 10.0
6-10
630.0
122
$20.00
Opes
VL-4K
125
600
037
BaassV X22
4334
400
140
176
Llmau
13.5
261
935
14
12117.11
71331
240
18
11.5
18.541
-220
33.50
-152
345
1228
Devan-V1
18
Dwaed VS
734
26.0
10
1.0
23
Uweed VLL
738
2035.00
03.1
12hue
-114
Claak Oye
0.303
12
-01
1155
110
1,0
Gesad: Pro
055
06
11
11
4U3
GPT4 Taabo
03614
01
04
0.5
23
28.77177
2671
255
3310619
220
140
4014
18
134
12
2616
18
10.0
LL
233
703
-208
4230113
3.5
Mesi (0M. V2.3PT
71860
122
2136
S1STA
-131
Minaky
351
Devend VLL
9185
SKUS
-100
-.10
-2.3
Druech
T5.1
12
1.00
2:34
Claske Sor
07
03
03
Clooa
EPT Ay
15:4
3177.0
22.9
011
1/2
Decuu VL Max
4040
03.5
335521
6.56
275
122
-11
10.4
JUM
18
15.
226
Hart
10
03.3
128
377
wcc.eeeddddd VE-4K
518.00
1061
Opea
435
1000
1233-0
1811e
12.9
-130
100
Maskey
12.0
145.00
270
1700
1935.
15700.0
1675
1208
53.50
373
12.0
12.87
3.00
41.5
31.30
1922
335
08.
0
1:771
236
24

---

Published as a conference paper at ICLR 2025
E MORE RELATIONSHIP BETWEEN VCR AND OTHER BENCHMARKS
The heatmap shown in Figure 6 provides a detailed view of the pairwise correlation between 23 different benchmarks used to evaluate 38 VLMs. The scores of these models on each benchmark were utilized to compute this correlation matrix. The color intensity and numerical values represent the degree of correlation, ranging from -1 (perfect negative correlation) to (perfect positive correlation), with warmer colors indicating higher positive correlations and cooler colors indicating weaker or negative correlations.
We observe that VCREN, HARD is markedly different from other benchmarks in the evaluation set. demonstrates minimal and even sometimes negative correlation with other benchmarks. This suggests that the skill set required for VCREN, HARd is largely unrelated those tested by other popular tasks emphasizing more straightforward image-to-text associations or OCR capabilities. Specifically, VCREN, HARD challenges models with tasks that involve high-level reasoning and mini- mal reliance on pixel-level information, focusing instead on understanding context, commonsense reasoning, and visual narrative interpretation. These features are less critical in other benchmarks, which explains the weak correlation across tasks. VCREN. EASY exhibits a slightly stronger correlation with few other benchmarks but remains moderately independent of most others. Like VCREN, HARD VCREN, EASY also evaluates visual commonsense reasoning, but with less stringent requirements, offering models more cues and simpler connections between visual elements and textual understand- ing. This leads to moderate overlap with benchmarks like Tex(VQA, which similarly focus on text understanding in a visual context, but VCREN, EASY still emphasizes a higher level of interpretative reasoning than standard text-based vision benchmarks.
Cormilatin Heatmas Sanchmaria
0C/0ench
COOUUL L
CORNAL TESTOOR
2
IPC-
POEM YOL JEGLAMMEE
EKC
GUr
GG
120
9/5
S
11A
611
140
-84
Geodkech
Reevee uec
wre PR.eeun
wnstmbd PES0
Figure 6: The heat map of benchmarks displays the correlation between the metric scores of 38 models for each benchmark pair.
25

---

Published as a conference paper at ICLR 2025
F POTENTIAL QA
What could be the possible reason that CogVLM performs well in VCR-WIKI series bench- marks? Many models we tested (DocOwl-L.5, Monkey, MiniCPM-V2.5, InternLM series, InternVL series) follow similar inference pipeline to adapt to high-resolution application scenarios:
1. An algorithm divides the input image into segments.
2. Each segment is encoded into tokens using a CILP-based image encoder.
3. A filtering mechanism (algortin/resampeerabbtttttt)))) processes the visual tokens.
4. The filtered tokens are concatenated with language tokens and input the LLM.
If, in step 3, pixel-level hints embedded in text within the image (TEL) are disregarded, the model cannot correctly answer the question. Consequently, some of these models may perform better on benchmarks emphasizing global features but struggle on the VCR- particularly in the hard partitions. For example, while Intern VL2-40B performs best on VCRgN, EASY. it does not perform well VCREN, HARD- As noted in the paper, the easy partition of the benchmark primarily verifies that the VCR task feasible for the models. In contrast, the hard partition explores the boundaries of VCR capability for both models and human test-takers (who require more time and focus to solve the puzzles in the hard partition).
series benchmarks,
The CL2 and Cambrian-l series, by contrast, do not include step in their inference pipelines. Instead, their image encoders operate at mid-to-high resolutions (IK level), and they resize the input image match the supported resolution rather than dividing into segments. The image encoder resolution for CogVLM2 is 1344 x 1344, while Cambrian-1 employs four image encoders, the largest supporting a resolution of 1024 x 1024. This approach may encounter challenges with extremely shaped input images (e.g.. 8192 1024). but for VCR-WIKI, where images are mostly near-square (on average 300 360 for VCRzi, ECERs/EAAY//V. EASY/VCRZiL,.... HARD and 300 375 for VCR EASY/V /VCr///RRnN.CCC haRd), high-resolution support is not necessary. For instance, IntemL VL outperforms XXmoolosMooooooe2--V --L-4KHDDD on this benchmark.
What could be the potential way to improve models' capability VCR? To suggest potential avenues for improving VLM performance on VCR, we propose the following:
Include VCR VLM Pretraining: Just as OCR parsing tasks are often included pre- training to improve OCR performance, researchers could consider incorporating VCR tasks during pretraining. We will codebase to facilitate this process, making as straightforward data augmentation.
2. Architectural Exploration: CogVLM2 is the best-performing model on average across the four partitions, and we believe this is largely due to its vision expert architecture. We contacted the CL2 team and learned that GLM-4 and Cog VLM share the same training data, yet there is a significant performance gap between them on the VCR benchmarks.
3. Chain-of-Thought (CoT) Methods: Researchers could explore multi-modality pipelines based on CoT techniques to improve existing VLMs on VCR tasks (Chen et al. 2024a Zhang al. 2023). Although model might not initially focus on the correct visual area (e.g., lexel-level hints in the TEI,, CoT-hased techniques could help refine its focus over successive rounds.
How is the Transferability of VCR-WIKI Finetuning? In Table 10 we show the transferability of VCR-WIKI by finetuning multiple models on different finetuning datasets' training sets and testing their performance a series of benchmarks.
The analysis of our experimental results highlights the strong transferability of the proposed VCR- WIKI dataset across various benchmarks. Notably, models fine-tuned on VCR-WIKI demonstrate significant performance improvements not only within the VCR-WIKI benchmarks themselves, but also across different language settings. For example, fine-tuning VLMV on VCREN. HaRd leads to a substantial increase in performance on the VCRzH, EASY benchmark, elevating the score from 9.15 to 42.55. Similarly, fine-tuning the Chinese version of VLM2 VCRzi, HARD enhances its
26

---

Published as a conference paper at ICLR 2025
performance both the VCREN, EASY and VCREN, HARd benchmarks, with scores rising from 79.9 to 87.57 and from 25.13 to 44.97, respectively. These enhancements indicate that the VCR-wiki dataset facilitates the learning of effective transferable features even when the fine-tuning and evaluation involve different languages.
Additionally, the consistent achievement of the highest scores within each model's finetuning vari- ations underscores the robustness VCR-WIKI in improving model performance across diverse evaluation metries. This evidence collectively demonstrates that VCR-WIKI serves versatile and powerful resource for enhancing model generalization and performance across multiple tasks and linguistic contexts.
Table 10: Performance Comparison of Base Models.
CagVEMI
Fastesing.fatect
Neac
DKVVO Triis
VCRaLism
Noec
OKVON-Taas
VCRos sup
Nox:
OKVOA Trui
VCRO ESAR
77.43
12.30
16.55
75.45
1047
11.81
19:51
18.66
38.10
3135
79
03.27
7990
10.55
$3.57
27-22
VCR
410
3:33
22.55
VT
316
0.1
136
1.1
510
L/0
111
133
13.11
MaSter
T65
11.6
3156
7330
7240
MME
30316
1996
183105
1073
1868.
1981
188221
38017
186
1998
7699
OCR BESCHI
71.30
11.00
71.00
11.30
71:00
72.81
79.8
77.91
75330
71.68
45.75
56.90
36.33
CherQa
71.30
72.17
72.33
73.0
SUR
79
03.16
19.00
65.4
27

---

Published as a conference paper at ICLR 2025
G REAL-WORLD CASE STUDY OF MODELS FINE-TUNED ON VCR-WIKI
This case study aims to assess the real-world applicability of models fine-tuned on VCR-WIKI for recognizing occluded text, a challenging task with significant practical implications. The setting involves evaluating the performance of three state-of-the-ar models, namely MiniCPM-V2.5 8B, CogVLM2 19B, and Qwen2- VL 7B, on curated dataset of eleven photographs featuring occluded text from real-world scenarios, such as street maps and collected images. Since no standard benchmark exists for real-world occluded text recognition, this dataset serves as a proxy to measure the efficacy of VCR fine-tuning in improving performance. We show whether the model completely recovers the occluded or distorted texts in the image with (correct) or x (partially correct or incorrect). This evaluation provides insight into how VCR fine-tuning translates to practical challenges and complements the quantitative analyses presented in the main paper.
BANK OF FAMERICA
Figure 7: Ground-truth: BANK OF AMERICA TWO BRYANT PARK
Figure 8: Ground-truth: METROPOLITAN
Figure 9: Ground-truth:
NOT IN SERVICE
MiniCPM: /
MiniCPM:
MiniCPM:
MiniCPM-ft: V
MiniCPM-ft:
MiniCPM-ft: V
CogVLM2: V
CogVLM2:
CogVLM2: V
CogVLM2-ft: V
CogVLM2-ft:
C0gVLM2-ft: V
Qwen2-VL-7BB VL-7B:
Qwen2-VL-7B: VL-7B: /
Qwen2- VL-7B: V
Qwen2-VL--B-ft:
Qwen2-VL-7B-ft:
Qwen2 VL-7B-ft:
Hiee Peapack
Figure 10: Ground-truth: NO CYCLING
Figure 11: Ground-truth: SHIPPING FAX SERVICE
Figure 12: Ground-truth Home of Peapack Private
PASSPORT PHOTOS COMPUTER RENTALS
MiniCPM: x
MiniCPM: /
MiniCPM-ft:
MiniCPM-ft: /
MiniCPM: x
CogVLM2: /
CogVLM2: V
MiniCPM-ft: X
CMgVLMM2ftt: V
CogVI.M2-ft: /
CogVLM2: V
Qwen2- VL.-B: X
Qwen2-VI--77
CogVLM2-ft: V
Twen2-VL-7B-. v
Qwen2-VL-7B.. /
Qwen2- VL-7B: /
-e. /
28

---

Published as a conference paper at ICLR 2025
$24.14
AMOU TF
90
$27
TOTAL
EED
UPTN RESEFT
DELICCY
AMERACAM AN
ESS
00
00
BREAKFAST LUNCH CATERING
THANK
NER
Figure 13: Ground-truth: DELICACY BREAKFAST LUNCH CATERING
Figure 14: Ground-truth: INSPECT UPON RECEIPT...
Figure 15: Ground-truth: AMOUNT TOTAL APPROVED AMERICAN N EXPRESS AID THANK YOU MERCI
DO NOT SIGN...
VISIBLE DAMAGE?
MiniCPM: v
CUSTOMER COPY
MiniCPM: x
MiniCPM-ft: V
MiniCPM:
MiniCPM-ft:
C22V V
MiniCPM-ft: v
CogVLM2: V
C2gVLM2-ft: V
CogVLM2: V
Cog V
Qwen2- VL-7B: V
CogVLM2-ft:
Dwen2-VL-7BB v
Qwen2-VL. VL-7B-ft:
Owen2-VL-7B-HE
Qwen2- VL-7B-ft: V
mey
See uot
OUTDOOR ADVERTISING
Three American 1969
Figure 16: Ground-truthi INSPECT UPON RECEIPT...
Figure 17: Ground-trutd AMOUNT TIP TOTAL APPROVED AMERICAN EXPRESS AID THANK MMECI CUSTOMER COPY
DO NOT SIGN... VISIBLE DAMAGE?
MiniCPM:x
MiniCPM: x
MiniCPM-ft:
MiniCPM-ft: V
CogVLM2: V
CogVLM2:
CogVLMM-ft:
CogVLM2-ft: /
Qwen2- VL-7B:
Qwen2-VL-7B. VL-7B: /
Qwen2- Owen2-VL-7B... VL-7B-ft: /
-wen2-VL-7B-fi- /
29


Reviews:
[{"review_id": "y7eIXHx0dg", "reviewer": "Reviewer_2nhH", "summary": "The paper introduces a new vision-language evaluation task (VCR), for assessing models performance on restoring partially obscured text embedded in images, using pixel-level hints (pixels from the top and the bottom of the partially obscured characters) and the visual contexts of the image. This approach mirrors human proficiency in recognising partially hidden objects, a skill supported by complex visual and cognitive processes, as noted in prior studies.\n\nThe authors claim, that the traditional VQA tasks, especially in the area of embedded text recognition, work independently the non-text visual context of the image, a gap that the VCR task covers. To support this claim, they are analysing the correlation of the VCR task to other VLM benchmarks and they demonstrate low correlation especial for the VCR-HARD task, where pixel hints are minimal.\n\nFor this topic, the authors developed a pipeline for generating synthetic images for the VCR task, using image-caption pairs, with configurable caption visibility to control the difficulty level of the task. Using this pipeline they developed the VCR-wiki dataset, from images and captions coming from Wikipedia. The dataset contains 2.11M English and 346K Chinese samples. There are two versions of the dataset, an EASY one and a HARD one (both feasible for native speakers of the language), where these two versions differ on the amount of the pixels remain visible from the obscured text. Each dataset has a Train, a Validation and a Test split.\n\nFor the evaluation task the authors introduce two metrics, an Exact Match, and Jaccard Index (Bag-of-Words as sets) for comparing the ground truth (obscured text) with the models' response (recovery of the obscured text).\n\nThe authors include in the paper several VCR evaluation experiments for many open and closed source models. They show that the majority of the models performs poorly on this task, and they draw the below conclusions:\n* VCR is a challenging task for SOTA VLMs, while humans score high in this task\n* Models that perform well in OCR tasks, can score low in VCR\n* Performance degradation on Chinese version of the task, showing the need of improvements in multilingual capabilities\n* Larger models, or higher image resolutions are not correlated with better performance in VCR task\n* Non-text image information is not effectively leveraged by the most of the models in this task\n* Model architecture can be a performance improvements resisting factor when applying fine-tuning on VCR-wiki dataset", "questions": "Quoted from the paper: \"Figure[3] is an example VCR task in hard mode, and Figure[1] shows an example VCR task in the easy\nmode. Although humans can still fill the blanks easily in the hard mode, it is nearly impossible for models with only OCR capabilities to recover the covered texts without using the context.\" Instead of Figure[3] I think it should be Figure[2].\n\nSuggestions:\nAddress the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper introduces an original task for evaluating VLMs. The contributions are clearly explained, and there are sufficient details about them (metrics, pipeline for generating synthetic images for the VCR task, VCR-wiki dataset). The experiments are presented with clarity and the paper has a good structure.", "weaknesses": "The paper lacks a clear explanation of how improvements in VLMs, as reflected by higher scores on VCR task, would translate to overall advancements in VLM capabilities. There are not any specific ways mentioned in which enhanced performance on this task would lead to meaningful or practical improvements in VLM functionality, generalisation, or real-world applicability. Providing a more detailed rationale on this point would strengthen the paper, as it would help clarify the relevance of VCR task performance as a benchmark for broader VLM development.\n\nThere are additional questions regarding the choices made in various experimental settings, and providing more details on these decisions would strengthen the clarity of the paper. Specifically:\n* Selected number of n-grams: The authors chose to use a 5-gram approach; however, it is not clear why this number was selected. Was this choice based on empirical results, or was it simply an arbitrary decision? Additionally, it would be informative to know if smaller n-grams were tested and, if so, how the models performed with those alternative configurations.\n* Text visibility settings: For the \"EASY\" mode in the dataset, how did the authors determine the chosen visibility threshold? Was this the minimum level at which open-source OCR tools started to perform poorly, or was it a randomly selected setting? A discussion on how this visibility threshold was established would provide useful context, especially regarding the task's difficulty calibration.\n* Fine-tuning configuration: Given the extensive data available in the VCR-wiki dataset, it would be helpful to understand if the authors experimented with fine-tuning the models using a larger subset of this data. Additionally, specifying the number of training epochs used in these experiments would offer valuable insight into the training process and potential impacts on model performance.\n* Prompts: It would be beneficial to know if any prompt engineering was applied when experimenting with different models. For instance, how did the authors arrive at the chosen prompt? In relation to the observed performance degradation with VI, did the authors attempt prompts specifically guiding the model to utilize visual cues (e.g., \"Use the visual information from the image to recover the missing text.\")? \n\nAddressing these points with more detail would enhance the paper’s transparency and rationale behind these experimental choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1146wUVFMJ", "reviewer": "Reviewer_kwun", "summary": "The paper introduces the Visual Caption Restoration (VCR) benchmark, designed to assess vision-language models' (VLMs) ability to restore partially obscured text using visual cues from images. VCR is built with images and captions sourced from Wikipedia, encompassing 2.11M English and 346K Chinese entries, presented in both easy and hard configurations. Experiments are conducted on state-of-the-art (SOTA) VLMs to evaluate their performance.", "questions": "* As mentioned above, my main question centered around the details about the dataset:\n  * Given that training data for SOTA VLMs often includes Wikipedia content, how does the benchmark ensure that images and captions in the dataset have not been seen during VLM pre-training or SFT? When were these images and captions posted?\n  * The difference between easy and hard cases is vague.Separating these cases based solely on the area of visible pixels may be insufficient, as the difficulty level for restoring a word can also depend on the word itself. For example, if both \"area\" and \"idea\" are 80% covered, it may be challenging to determine which is easier to recognize. The word \"idea\" initially occupies more pixels, and the \"d\" token could serve as a clue, making it more recognizable. This suggests that the difficulty of a sample may also depend on word combinations in the sentence. What are the detailed criteria (e.g. covering proportion) for choosing easy/hard samples? \n  * Following up on the previous point, if the covering proportion is automatically used to select easy and hard samples, would it be possible to manually curate these samples based on word combinations? I understand that this would involve additional effort, so this is just an open question I'm interested in.\n\n\n\n---------\nPost rebuttal: \n\n\nHaving reviewed all the reviews and rebuttals again, I have decided to raise my rating. I also encourage the authors to continue exploring and developing this promising topic.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and easy to follow.\n* VCR is an interesting benchmark that poses challenges to existing VLMs. It's a nice addition to the VLM community.\n* The evaluation of most SOTA VLMs offers insightful findings that advance the field.", "weaknesses": "* Certain important details about the dataset are missing from the paper; please refer to the questions section below.\n* The experiments on fine-tuning VLMs on VCR are not informative enough, as the VLMs of choice differ too much in terms of backbone/architecture/training pipeline... I feel like stating \"the architectural design of CogVLM2 is well-suited for VCR\" in line 368 might need more experiments or well-controlled experimental setups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vptu4aDXH1", "reviewer": "Reviewer_L8oC", "summary": "This paper presents Visual Caption Restoration (VCR), a new VL task where models need to accurately restore partially obscured text embedded within images using pixel-level hints and contextual cues. Unlike conventional OCR tasks, VCR requires the model to align vision and language modalities to restore text that is often obscured or fragmented. The authors introduce VCR-WIKI, a large-scale synthetic dataset generated using Wikipedia captions, supporting both English and Chinese text in easy and hard configurations. Experiments demonstrate that state-of-the-art vision-language models fall significantly short of human performance on the VCR task, highlighting the need for new model architectures that handle complex intermodal alignment.", "questions": "Since VCR-WIKI relies on synthetic captions and partially obscured text, I was wondering to what extent you expect models trained on this dataset to generalize to real-world scenarios where text may have more complex occlusions or be influenced by environmental factors (e.g., lighting, noise).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies a critical gap in vision-language tasks by focusing on restoring partially occluded text within images. This differs from traditional VQA and OCR by challenging models to reconstruct text based on incomplete visual information and context, a complex multimodal problem.\n\nThe VCR-WIKI dataset provides a scalable and controllable environment for testing models under different levels of text occlusion. The authors’ pipeline generates a diverse dataset that is balanced across languages, allowing for cross-linguistic studies and detailed analysis of model capabilities.\n\nBy analyzing the model failures, the paper provides useful insights into the challenges of multimodal alignment, reasoning, and generalization. The authors' discussion on how models with strong OCR capabilities underperform on VCR highlights the distinct challenges VCR poses.", "weaknesses": "The reliance on synthetically generated images raises questions about the generalizability of models trained on VCR-WIKI to real-world applications. Text within real images can exhibit more variability in terms of style, font, and environmental factors that may not be fully captured by VCR-WIKI.\n\nThe paper's use of GPT-based metrics could introduce variability in assessment, especially given the subjective nature of scoring masked text restoration. While Exact Match and Jaccard Index provide quantitative metrics, more robust evaluation strategies would be beneficial to standardize results across future works.\n\nThe synthetic dataset may not fully replicate the nuances of real-world images with natural text occlusion. Additional discussion on the challenges of transferring VCR models to real-world scenarios would clarify the dataset's practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)