VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text

Authors: Tianyu Zhang, Suyuchen Wang, Lu Li, Ge Zhang, Perouz Taslakian, Sai Rajeswar, Jie Fu, Bang Liu, Yoshua Bengio
Primary area: datasets and benchmarks
Keywords: dataset, vision-language model, multimodal

Abstract:
We introduce Visual Caption Restoration (VCR), a novel vision-language task that challenges models to accurately restore partially obscured texts using pixel-level hints within images through complex reasoning. This task stems from the observation that text embedded in images intrinsically differs from common visual elements and text due to the need to align the modalities of vision, text, and text embedded in images. While many works incorporate text into images for visual question answering, they mostly rely on OCR or masked language modeling, reducing the task to text-based processing. However, text-based processing becomes ineffective in VCR as accurate text restoration depends on the combined information from provided images, context, and subtle cues from the tiny, exposed areas of masked texts. We develop a pipeline to generate synthetic images for the VCR task using image-caption pairs, with adjustable caption visibility to control the task difficulty. With this pipeline, we construct VCR-WIKI for VCR using Wikipedia images with captions, including 2.11M English and 346K Chinese training entities, plus 5K validation and 5K test entities in both languages, each in easy and hard configurations. We also make a hidden test set, VCR-HIDDEN, to avoid potential overfitting on VCR-WIKI. Our results reveal that current vision-language models significantly lag behind human performance in the VCR task, and merely fine-tuning the models on our dataset does not lead to notable improvements. We release VCR-WIKI and the data construction code to facilitate future research.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730429646765

thread_type:
official_review

<thread_object>
creator: reviewer_l8oc

<content>
summary:
This paper presents Visual Caption Restoration (VCR), a new VL task where models need to accurately restore partially obscured text embedded within images using pixel-level hints and contextual cues. Unlike conventional OCR tasks, VCR requires the model to align vision and language modalities to restore text that is often obscured or fragmented. The authors introduce VCR-WIKI, a large-scale synthetic dataset generated using Wikipedia captions, supporting both English and Chinese text in easy and hard configurations. Experiments demonstrate that state-of-the-art vision-language models fall significantly short of human performance on the VCR task, highlighting the need for new model architectures that handle complex intermodal alignment.

strengths:
The paper identifies a critical gap in vision-language tasks by focusing on restoring partially occluded text within images. This differs from traditional VQA and OCR by challenging models to reconstruct text based on incomplete visual information and context, a complex multimodal problem.

The VCR-WIKI dataset provides a scalable and controllable environment for testing models under different levels of text occlusion. The authors’ pipeline generates a diverse dataset that is balanced across languages, allowing for cross-linguistic studies and detailed analysis of model capabilities.

By analyzing the model failures, the paper provides useful insights into the challenges of multimodal alignment, reasoning, and generalization. The authors' discussion on how models with strong OCR capabilities underperform on VCR highlights the distinct challenges VCR poses.

weaknesses:
The reliance on synthetically generated images raises questions about the generalizability of models trained on VCR-WIKI to real-world applications. Text within real images can exhibit more variability in terms of style, font, and environmental factors that may not be fully captured by VCR-WIKI.

The paper's use of GPT-based metrics could introduce variability in assessment, especially given the subjective nature of scoring masked text restoration. While Exact Match and Jaccard Index provide quantitative metrics, more robust evaluation strategies would be beneficial to standardize results across future works.

The synthetic dataset may not fully replicate the nuances of real-world images with natural text occlusion. Additional discussion on the challenges of transferring VCR models to real-world scenarios would clarify the dataset's practical applicability.

questions:
Since VCR-WIKI relies on synthetic captions and partially obscured text, I was wondering to what extent you expect models trained on this dataset to generalize to real-world scenarios where text may have more complex occlusions or be influenced by environmental factors (e.g., lighting, noise).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498985799

thread_type:
official_review

<thread_object>
creator: reviewer_kwun

<content>
summary:
The paper introduces the Visual Caption Restoration (VCR) benchmark, designed to assess vision-language models' (VLMs) ability to restore partially obscured text using visual cues from images. VCR is built with images and captions sourced from Wikipedia, encompassing 2.11M English and 346K Chinese entries, presented in both easy and hard configurations. Experiments are conducted on state-of-the-art (SOTA) VLMs to evaluate their performance.

strengths:
* The paper is well-written and easy to follow.
* VCR is an interesting benchmark that poses challenges to existing VLMs. It's a nice addition to the VLM community.
* The evaluation of most SOTA VLMs offers insightful findings that advance the field.

weaknesses:
* Certain important details about the dataset are missing from the paper; please refer to the questions section below.
* The experiments on fine-tuning VLMs on VCR are not informative enough, as the VLMs of choice differ too much in terms of backbone/architecture/training pipeline... I feel like stating "the architectural design of CogVLM2 is well-suited for VCR" in line 368 might need more experiments or well-controlled experimental setups.

questions:
* As mentioned above, my main question centered around the details about the dataset:
  * Given that training data for SOTA VLMs often includes Wikipedia content, how does the benchmark ensure that images and captions in the dataset have not been seen during VLM pre-training or SFT? When were these images and captions posted?
  * The difference between easy and hard cases is vague.Separating these cases based solely on the area of visible pixels may be insufficient, as the difficulty level for restoring a word can also depend on the word itself. For example, if both "area" and "idea" are 80% covered, it may be challenging to determine which is easier to recognize. The word "idea" initially occupies more pixels, and the "d" token could serve as a clue, making it more recognizable. This suggests that the difficulty of a sample may also depend on word combinations in the sentence. What are the detailed criteria (e.g. covering proportion) for choosing easy/hard samples? 
  * Following up on the previous point, if the covering proportion is automatically used to select easy and hard samples, would it be possible to manually curate these samples based on word combinations? I understand that this would involve additional effort, so this is just an open question I'm interested in.



---------
Post rebuttal: 


Having reviewed all the reviews and rebuttals again, I have decided to raise my rating. I also encourage the authors to continue exploring and developing this promising topic.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730861534773

thread_type:
official_review

<thread_object>
creator: reviewer_2nhh

<content>
summary:
The paper introduces a new vision-language evaluation task (VCR), for assessing models performance on restoring partially obscured text embedded in images, using pixel-level hints (pixels from the top and the bottom of the partially obscured characters) and the visual contexts of the image. This approach mirrors human proficiency in recognising partially hidden objects, a skill supported by complex visual and cognitive processes, as noted in prior studies.

The authors claim, that the traditional VQA tasks, especially in the area of embedded text recognition, work independently the non-text visual context of the image, a gap that the VCR task covers. To support this claim, they are analysing the correlation of the VCR task to other VLM benchmarks and they demonstrate low correlation especial for the VCR-HARD task, where pixel hints are minimal.

For this topic, the authors developed a pipeline for generating synthetic images for the VCR task, using image-caption pairs, with configurable caption visibility to control the difficulty level of the task. Using this pipeline they developed the VCR-wiki dataset, from images and captions coming from Wikipedia. The dataset contains 2.11M English and 346K Chinese samples. There are two versions of the dataset, an EASY one and a HARD one (both feasible for native speakers of the language), where these two versions differ on the amount of the pixels remain visible from the obscured text. Each dataset has a Train, a Validation and a Test split.

For the evaluation task the authors introduce two metrics, an Exact Match, and Jaccard Index (Bag-of-Words as sets) for comparing the ground truth (obscured text) with the models' response (recovery of the obscured text).

The authors include in the paper several VCR evaluation experiments for many open and closed source models. They show that the majority of the models performs poorly on this task, and they draw the below conclusions:
* VCR is a challenging task for SOTA VLMs, while humans score high in this task
* Models that perform well in OCR tasks, can score low in VCR
* Performance degradation on Chinese version of the task, showing the need of improvements in multilingual capabilities
* Larger models, or higher image resolutions are not correlated with better performance in VCR task
* Non-text image information is not effectively leveraged by the most of the models in this task
* Model architecture can be a performance improvements resisting factor when applying fine-tuning on VCR-wiki dataset

strengths:
The paper introduces an original task for evaluating VLMs. The contributions are clearly explained, and there are sufficient details about them (metrics, pipeline for generating synthetic images for the VCR task, VCR-wiki dataset). The experiments are presented with clarity and the paper has a good structure.

weaknesses:
The paper lacks a clear explanation of how improvements in VLMs, as reflected by higher scores on VCR task, would translate to overall advancements in VLM capabilities. There are not any specific ways mentioned in which enhanced performance on this task would lead to meaningful or practical improvements in VLM functionality, generalisation, or real-world applicability. Providing a more detailed rationale on this point would strengthen the paper, as it would help clarify the relevance of VCR task performance as a benchmark for broader VLM development.

There are additional questions regarding the choices made in various experimental settings, and providing more details on these decisions would strengthen the clarity of the paper. Specifically:
* Selected number of n-grams: The authors chose to use a 5-gram approach; however, it is not clear why this number was selected. Was this choice based on empirical results, or was it simply an arbitrary decision? Additionally, it would be informative to know if smaller n-grams were tested and, if so, how the models performed with those alternative configurations.
* Text visibility settings: For the "EASY" mode in the dataset, how did the authors determine the chosen visibility threshold? Was this the minimum level at which open-source OCR tools started to perform poorly, or was it a randomly selected setting? A discussion on how this visibility threshold was established would provide useful context, especially regarding the task's difficulty calibration.
* Fine-tuning configuration: Given the extensive data available in the VCR-wiki dataset, it would be helpful to understand if the authors experimented with fine-tuning the models using a larger subset of this data. Additionally, specifying the number of training epochs used in these experiments would offer valuable insight into the training process and potential impacts on model performance.
* Prompts: It would be beneficial to know if any prompt engineering was applied when experimenting with different models. For instance, how did the authors arrive at the chosen prompt? In relation to the observed performance degradation with VI, did the authors attempt prompts specifically guiding the model to utilize visual cues (e.g., "Use the visual information from the image to recover the missing text.")? 

Addressing these points with more detail would enhance the paper’s transparency and rationale behind these experimental choices.

questions:
Quoted from the paper: "Figure[3] is an example VCR task in hard mode, and Figure[1] shows an example VCR task in the easy
mode. Although humans can still fill the blanks easily in the hard mode, it is nearly impossible for models with only OCR capabilities to recover the covered texts without using the context." Instead of Figure[3] I think it should be Figure[2].

Suggestions:
Address the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732559929574

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
TL;DR: Summary of the Rebuttal

comment:
We sincerely thank you for your thorough and insightful feedback on our submission. Your constructive comments have been invaluable in refining our work, and we have addressed each concern meticulously in our detailed rebuttals. Below, we provide a concise summary of our responses, underscoring the significance and contributions of our paper:

* **Enhanced Evaluation and Transferability** (Reviewers 2nhH & L8oC):

  * **Comprehensive Fine-Tuning Results**: We conducted extensive experiments demonstrating that fine-tuning on the VCR-wiki dataset not only improves performance on the VCR task but also enhances model capabilities across various benchmarks, including OCR-Bench and real-world scenarios. These findings, detailed in Appendices D and E, highlight the dataset's robustness and practical applicability.
  * **Real-World Applicability**: Through a case study with real-world occluded text images, we showcased significant performance improvements post fine-tuning, affirming the dataset's relevance beyond synthetic benchmarks.

* **Dataset Design and Masking Strategy** (Reviewers 2nhH & kwun):

  * **Rationale for 5-gram Masking**: We clarified that the 5-gram masking balances linguistic meaningfulness and   empirical feasibility, capturing essential grammatical structures while maintaining task difficulty. This approach ensures the task remains challenging yet attainable.
  * **Distinct Easy and Hard Settings**: Both settings utilize the same data with varying degrees of pixel masking, ensuring direct comparability. This design choice was emphasized to clarify the dataset's structure and the controlled difficulty levels. The fully controlled easy and hard settings, VI+TEI and VI settings, and the "finetune-ability" experiment between pre- and post-finetuning provides directly comparible results for directed guidance for future VLM developments.

* **Fine-Tuning Protocol and Experimental Setup** (Reviewers 2nhH & kwun):

  * **Methodological Clarity**: We provided detailed explanations of our fine-tuning procedures, including the selection of a representative subset and the decision to use a single, standardized prompt. This ensures reproducibility and consistency across evaluations.
  * **Resource Considerations**: Acknowledging resource constraints, we emphasized that our fine-tuning experiments effectively highlight diverse model behaviors, laying the groundwork for future, more extensive studies.

* **Generalizability and Real-World Applicability** (Reviewer L8oC):

  * **Case Study Inclusion**: By adding a real-world case study in Appendix E, we demonstrated tangible improvements in models' ability to interpret occluded text in practical settings, reinforcing the dataset's utility and the task's relevance.

* **Robustness of Evaluation Metrics** (Reviewer L8oC):

  * **Appropriate Metrics Selection**: We justified the use of Exact Match and Jaccard Index as robust, objective metrics tailored to the VCR task's requirements. This ensures reliable and consistent evaluation aligned with the task's objectives.

* **Additional Enhancements and Flexibility** (All Reviewers):

  * **Dataset Flexibility**: We highlighted the VCR framework's adaptability, allowing researchers to customize masking configurations and explore domain-specific challenges. This flexibility fosters broader research applications and future innovations.
  * **Codebase Improvements**: Incorporating suggestions, we enhanced our codebase to support more nuanced masking strategies, such as whitelisting and blacklisting specific words, paving the way for more refined and targeted studies.

In summary, VCR presents a **pioneering vision-language task that bridges the gap between visual cues and textual restoration**. Our meticulously designed dataset, coupled with comprehensive evaluations, reveals significant performance gaps in current models, underscoring the task's challenge and the dataset's value. Besides, the VCR-wiki dataset provides **fully comparable configurations** for difficulty, image layout and finetune-ability **for more targeted model improvements**. By releasing VCR-wiki and the accompanying code, we aim to catalyze future research, fostering advancements that move vision-language models closer to human-like multimodal understanding.

We are confident that our revisions and detailed responses address your concerns comprehensively, highlighting the paper's contributions and its potential impact on the field. We appreciate your consideration and hope that our efforts have clarified the strengths and significance of our work.

Thank you for your time and valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Visual Caption Restoration (VCR), a new vision-language task designed to evaluate how well models can restore partially obscured text within images using both pixel-level hints (visible parts of the characters) and the surrounding visual context. Unlike traditional OCR, VCR requires models to integrate visual and textual information to reconstruct fragmented or hidden text, mirroring human abilities. The authors created VCR-Wiki, a large-scale synthetic dataset derived from Wikipedia image-caption pairs, available in both English (2.11M samples) and Chinese (346K samples) and in "easy" and "hard" versions based on the degree of text obscuration.  Experiments are conducted on state-of-the-art (SOTA) VLMs to evaluate their performance.

The reviewers agree this is a novel and important task with well-defined methodology and dataset. The experiments and analysis are interesting and insightful.