Evaluating the Reading Ability of Large Multimodal Models
Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Jiuxiang Gu, Ryan A. Rossi, Changyou Chen


Reviewer paper summaries: This paper introduces the Multi-Modal Reading (MMR) benchmark, which aims to evaluate the reasoning and spatial understanding abilities of large language models (LMMs) in the context of text-rich image comprehension. The benchmark includes eleven diverse tasks designed to assess different aspects of multimodal reading abilities. These tasks span various subtasks such as text recognition, font recognition, element localization, spatial relationship understanding, object grounding, and text grounding. The dataset is derived from the LAION-5B dataset and augmented through human annotations and GPT-4V-generated question-answer pairs. The paper presents an evaluation of various state-of-the-art LMMs, revealing limitations and suggesting potential applications in real-world scenarios. The ultimate goal is to provide a more rigorous assessment framework that can better differentiate the performance of various LMMs and guide future research and development efforts in this domain.
Summary of strengths: One major strength of this paper is its contribution to the development of a new dataset specifically designed to test multimodal reading abilities in text-rich images. The MMR benchmark includes human-annotated data and question-answer pairs generated through advanced language models, which are then verified by human annotators. This combination aims to create a more reliable and comprehensive evaluation set that goes beyond simple information extraction tasks typically found in existing benchmarks. The paper's effort in generating a broad range of tasks—spanning from text recognition to complex spatial relationship understanding—stands out as a valuable resource for the research community.
Summary of weaknesses: 1. **Lack of Distinctiveness**: The new benchmark does not seem to offer significant innovation over existing benchmarks like DocVQA [1], Infographic VQA [2], and ChartQA [3]. Although the authors claim these existing benchmarks are primarily extraction-based, the tasks proposed in MMR also appear to align closely with extractive QA and are similar to tasks covered by other LMM benchmarks such as SEED series [4].
   
2. **Insufficient Comparative Analysis**: The paper does not adequately compare the MMR benchmark with existing benchmarks. It would strengthen the paper if it included a comparative analysis that shows how the MMR benchmark measures unique abilities or uncovers model weaknesses that other benchmarks do not capture.

3. **Annotation Process**: The process of annotation, especially the role of human annotators in verifying and correcting pseudo-labeled data from OCR APIs, is not sufficiently detailed. For a benchmark-focused paper, the reliability and transparency of the annotation process are crucial for credibility, and the current explanation is lacking.

4. **Prompting and Evaluation**: The paper uses a fixed prompt to evaluate various models, which raises concerns about bias. Different models may be optimized for different types of prompts, and using a single fixed prompt could result in skewed performance evaluations. The paper should consider the impact of prompt design and evaluation biases.

5. **Evaluation Metrics**: Despite claiming to assess reasoning abilities, the evaluation metrics are not notably innovative. Relying on normalized edit distance as a primary metric may not fully capture the complexity and reasoning capabilities that the paper aims to measure. The metrics should be more aligned with the purported goals of measuring advanced reasoning skills.

Reference
- [1] https://www.docvqa.org/
- [2] https://www.docvqa.org/datasets/infographicvqa
- [3] https://github.com/vis-nlp/ChartQA
- [4] https://github.com/AILab-CVC/SEED-Bench
Comments, suggestions and typos: typo:

- (line 76) text-ruch images -> text-rich images
Overall assessment scores: 2