Structured Attention Matters to Multimodal LLMs in Document Understanding

## Abstract

Abstract Document understanding remains a significant challenge for multimodal large language models (MLLMs). While previous research has primarily focused on locating evidence pages through precise multimodal queries, our work investigates a fundamental yet overlooked aspect: how input format influences document comprehension performance. Through systematic analysis, we discover that raw OCR text often impairs rather than improves MLLMs’ performance, which is a counterintuitive finding we attribute to attention dispersion and structure loss. To further substantiate our hypothesis, we propose a novel structure-preserving approach that encodes document elements using the LaTeX paradigm, maintaining the hierarchical organization and spatial relationships critical for comprehension. Our attention analysis reveals that structured text induces structured attention patterns on both textual and visual content, directing models to focus on semantically meaningful regions while reducing attention waste. This approach significantly enhances MLLMs’ document question answering performance across diverse document types without requiring architectural modifications or additional training.

## 1 Introduction

Document understanding is a common task in daily life. With the rapid development of multimodal large language models (MLLMs), capable general-purpose systems are increasingly expected to comprehend documents effectively (Zhu et al. (2023); Liu et al. (2024a, 2023); Wang et al. (2024a); Bai et al. (2023); Team (2025); Anthropic (2023); Dong et al. (2024); Chen et al. (2024b); Li et al. (2023); Wang et al. (2024b)). The difficulty in document question answering arises from the diverse nature of questions and the variety of information required, which evidence elements include text blocks, charts, diagrams, and figures, often requiring integration of multiple information sources.
Document understanding through MLLMs presents three key challenges: (1) the information diversity challenge of processing heterogeneous elements, (2) the context integration challenge of synthesizing scattered information, and (3) the structural relationship challenge of understanding how elements relate spatially and logically—relationships intuitive for humans but difficult for machines (Han et al. (2025)).

Previous research has addressed these challenges primarily by extending context windows to accommodate more content or by developing specialized architectures for extracting multi-granularity information (Ding et al. (2022); Hu et al. (2024); Ye et al. (2023); Liu et al. (2024b); Park et al. (2024); Tito et al. (2023)).
With general-purpose MLLMs, the trend has shifted toward retrieval-augmented generation (RAG), which locates relevant evidence and feeds it into models as images, OCR text, or both (Mishra et al. (2019); Suri et al. (2024); Park et al. (2024)). Despite these advances, a critical question remains unexplored: how does the format of input information, rather than merely its content—influence document understanding? Current approaches typically extract OCR text but discard critical structural information (Khattab and Zaharia (2020); Faysse et al. (2024)). This leads to a counterintuitive phenomenon we observed across multiple datasets: unstructured OCR text often degrades rather than enhances MLLM performance compared to using images alone (Cho et al. (2024); Deng et al. (2024); Han et al. (2025); Zhang et al. (2024a)). This observation motivated us to investigate the relationship between input structure and model comprehension. We discovered that information structure fundamentally shapes how MLLMs allocate attention across document elements. With unstructured text, models exhibit scattered attention patterns, wasting computational resources and struggling to identify relationships between elements.

Our study includes two main steps. First, we propose an approach to preserve the structure of OCR text. We input the evidence OCR text and images into an MLLM and prompt it to interpret the images with the help of the OCR text as much as possible. This step aims to maintain the structure of diagrams, tables, and texts. The layout information can be preserved in the form of text, which is helpful during the answer generation process. We input the images along with the LaTeX paradigm to generate answers, exploring the importance of document layout and the structure of sub-elements on the evidence page. We find that the accuracy of MLLMs in answering questions is significantly improved. Second, we analyze the attention distribution and transformation with different inputs, comparing the cases of using images alone and images combined with structured text as input. We find that attention is more focused when constrained by the structured input.

Overall, our findings show that the ability of general-purpose models in document understanding can be improved simply by changing the input format. Furthermore, the attention transformation between different inputs demonstrates that the structured attention brought by structured input is key to improving the ability of MLLMs to answer DocQAs. This indicates that there are other aspects worth exploring beyond focusing solely on enhancing the effectiveness of Retrieval Augmented Generation (RAG).

Figure: Figure 2: Structured text is generated using LaTeX. We prompt the MLLM to capture the layout of the given images as accurately as possible, producing blocks that include text, charts, and data tables. Figures that cannot be parsed into concrete content are represented using virtual paths and the LaTeX paradigm. This approach is simple and only requires API-level access with instruction-level control.
Refer to caption: x2.png

## 2 Related Work

We discuss two lines of related work: MLLMs in DocQA tasks and methods used in DocQA tasks.

#### MLLMs in DocQA Tasks.

Recent Document Visual Question Answering (DocVQA) models focus on expanding context window sizes, improving fine-grained visual comprehension, and enhancing layout analysis (Hu et al. (2024); Liu et al. (2024b); Tworkowski et al. (2023)). These systems often boost performance through architectural enhancements (e.g., added modules) or multi-stage training pipelines with diverse data inputs (Park et al. (2024); Tito et al. (2023); Chen et al. (2023)). While general multimodal models like Gemini (Team et al. (2024)) and Qwen-VL (Bai et al. (2023)) show improved visual processing and context handling, they remain constrained by input length limitations and multi-page processing capabilities. Their reliance on image-derived representations also hinders precise localization of detailed elements, while layout extraction from visual data risks diverting the model’s focus, reducing answer accuracy (Han et al. (2025)).

#### Methods in DocQA Tasks.

Retrieval augmentation has become pivotal for adapting general multimodal models to document question answering, addressing input constraints and information overload by retrieving fine-grained evidence through multimodal or unimodal queries (Lewis et al. (2020); Gao et al. (2023); Chen et al. (2024a); Cho et al. (2024)). While current methods enhance accuracy through optimized retrieval strategies (Deng et al. (2024); Lu et al. (2024)), input formatting remains challenging: image-only inputs hinder textual comprehension due to incomplete OCR integration, whereas text-only inputs (via OCR) neglect visual elements critical for image-dependent questions. Multimodal inputs combining text and images risk overwhelming models with redundant or conflicting data, leading to attention dispersion. To maximize performance, refining input strategies—such as modality prioritization, structured evidence fusion, or dynamic filtering—is essential for balancing information richness and focused reasoning in retrieved evidence.

## 3 Methodology

Understanding documents through multimodal large language models (MLLMs) represents a complex challenge at the intersection of vision and language processing. Unlike previous approaches that focus primarily on evidence retrieval or context window expansion, our research investigates how input representation fundamentally shapes model comprehension. In this section, we introduce a novel structure-preserving
approach that significantly enhances MLLMs’ document understanding capabilities without requiring architectural modifications or additional training. We analyze the attention transformation under different inputs, revealing that structured text induces structured attention patterns on both textual and visual content, enabling the comprehension of MLLMs.

### 3.1 Input Representation in Document Understanding

Document understanding requires models to interpret diverse information types—text blocks, tables, figures, charts, and their interrelationships. Through systematic analysis of MLLM performance in document question answering (DocQA) tasks, we identified a critical paradox

###### Observation 1 .

Providing unstructured OCR text alongside document images often degrades MLLM performance across benchmark datasets, despite increasing the total information available to the model.

This counterintuitive finding contradicts the common assumption in retrieval-augmented generation that more textual context improves performance. To investigate this phenomenon, we conducted a series of controlled experiments comparing MLLMs’ attention patterns when processing: (1) document images alone, (2) images with unstructured OCR text, and (3) images with our proposed structured text representation.

###### Observation 1 .

### 3.2 OCR text with structure

Based on observation [1](https://arxiv.org/html/2506.21600v1#Thmobservation1), we propose an approach to preserve the structure of the OCR results. As shown in [Figure 2](https://arxiv.org/html/2506.21600v1#S1.F2), we use LaTeX to encapsulate the OCR text. It is easy for MLLMs to understand LaTeX code because it usually follows a fixed and easy-to-understand paradigm. The LaTeX code constrains the content in titles, tables, and figures, providing efficient and additional references when MLLMs answer DocQAs with images. We input the image and the corresponding OCR text into the MLLM to obtain the structured code. We prompt the MLLM to preserve the structure and text related to charts in the image as much as possible. If a figure in the image cannot be converted into LaTeX content, we instruct the MLLM to use a virtual path to represent the figure while keeping the structure intact. An example is provided in [Figure 2](https://arxiv.org/html/2506.21600v1#S1.F2).

We use the proposed method to obtain structured text and compare the performance of MLLMs under three cases: images only, images with OCR text, and images with structured text. The results confirm the hypothesis in [Section 3.1](https://arxiv.org/html/2506.21600v1#S3.SS1), leading to the following observation:

Figure: Figure 3: MLLMs are less sensitive to image border tokens when constrained by structured text. The attention distribution shows that attention scores at the borders are lower in the presence of structured text.
Refer to caption: x3.png

###### Observation 2 .

Structured text and images work together to improve MLLMs’ performance in answering DocQAs.

Observation [2](https://arxiv.org/html/2506.21600v1#Thmobservation2) shows that the performance of MLLMs are improved significantly simply by preserving the structure of the input text, without the need to provide additional fine-grained positional information to help answer the questions. This observation proves the importance of structure, which means that merely informing the MLLMs of the structure of document helps them gain a better overview and answer questions correctly.

###### Observation 2 .

### 3.3 Attention transformation with structured OCR text

Given observation [2](https://arxiv.org/html/2506.21600v1#Thmobservation2), this subsection seeks to uncover the underlying reasons why structured textual content matters by analyzing the attention distribution when MLLMs answer questions using only images versus using structured text as references. [Figure 3](https://arxiv.org/html/2506.21600v1#S3.F3) shows that in the structured case, the attention scores are less sensitive to the boundaries of the image and more concentrated on the main body, indicating that the MLLMs know where to focus under the constraints of structured text. Based on these findings and experimental results, we have the following observation:

###### Observation 3 .

Structured text brings structured attention to both texts and images, which directly improves the abilities of MLLMs. This shows that structured attention is the key to helping MLLMs answer DocQAs.

Observation [3](https://arxiv.org/html/2506.21600v1#Thmobservation3) demonstrates the necessity of structure in document understanding. As shown in the example in [Figure 5](https://arxiv.org/html/2506.21600v1#S4.F5), structured text constrains MLLMs and reduces irrelevant attention when focusing on images. This helps MLLMs recover distracted attention and focus on images and relevant text to answer the given question.

Figure: Figure 4: A comparison of generated answers from MLLMs using OCR text versus structured text shows that structured text improves MLLM performance on DocQA tasks. The LaTeX paradigm helps preserve the image’s structure, aiding the model in locating evidence relevant to the question.
Refer to caption: x4.png

###### Observation 3 .

## 4 Experiments

We conduct evaluations on four document understanding benchmarks covering multiple scenarios to provide solid evidence supporting the observations presented in this paper.

### 4.1 Datasets and models

#### Implementation Details.

We apply the document preprocessing method from Han et al. (2025) to obtain textual content extracted using a combination of Optical Character Recognition (OCR) and PDF parsing techniques, and visual content by transforming pages in long documents into images. We locate the evidence pages and apply OCR to the corresponding images to obtain OCR results, then use the approach presented in [Section 3.2](https://arxiv.org/html/2506.21600v1#S3.SS2) to obtain the structured text. All experiments are conducted on 4 NVIDIA A100 GPUs.

#### Models.

Our evaluation uses the following four multimodal LLMs: Qwen2-VL-7B-Instruct, Qwen2.5-VL-7B-Instruct, LLaVA-v1.6-Mistral-7B and Phi-3.5-Vision-Instruct. These models accept both images and text as input and generate answers and analyses for the given questions.

#### Datasets.

The benchmarks include MMLongBench Ma et al. (2024), LongDocUrl Deng et al. (2024), PaperTab Hui et al. (2024), and FetaTab Hui et al. (2024). These evaluation datasets cover a variety of scenarios, including open- and closed-domain, textual and visual, as well as long and short documents, ensuring fairness and completeness in the evaluation.

#### Metrics.

For all benchmarks, we leverage Llama-3.1-8B-Instruct as the evaluation model to assess the consistency between the model’s output and the reference answer producing a binary decision (correct / incorrect). We report the average accuracy for each benchmark.

Figure: Figure 5: An example of attention transformation under two conditions: images alone versus images combined with structured text. The MLLMs tend to focus on text blocks, figures, and charts, rather than the overall layout.
Refer to caption: x5.png

### 4.2 Impact of OCR texts

We first compare the performance of MLLMs when using images alone versus using images with the help of OCR text. According to [Table 1](https://arxiv.org/html/2506.21600v1#S4.T1), the results show that OCR text impairs MLLMs’ performance on MMLongBench, which contains multi-element evidence images including charts, tables, blocks of text, and figures. The additional text information decreases the accuracy of the large model on this dataset from 0.389 to 0.370. The accuracy of MLLMs on other datasets might increase, but this can also impair their ability in specific cases. Providing unstructured OCR text alongside document images sometimes degrades MLLM performance across benchmark datasets, despite increasing the total information available to the model. These results and the case shown in [Figure 4](https://arxiv.org/html/2506.21600v1#S3.F4) clearly and convincingly support observation [1](https://arxiv.org/html/2506.21600v1#Thmobservation1), showing that repeated and redundant text information does not help large models answer questions about long documents.

Table: Table 1: Our proposed structure-preserving method effectively retains the document structure and significantly improves the accuracy of DocQA responses. For the open-domain dataset, we followed the retrieval method used in Han et al. (2025), using the top-1 retrieval result as input. On the closed-domain dataset, we used the evidence pages of the dataset as input to test the importance of document structure.

Table: Table 2: Performance comparison on different datasets. The ablation results show the importance of combining structured text and image. When structured text is combined with an image, it results in structured attention, which eventually helps MLLMs answer questions.

Table: Table 3: Performance comparison across different evidence sources on MMLongBench reveals how various document elements demonstrate the differing effects of structured input on document understanding.

### 4.3 Performance improvements with structured text

We consider the impact of structure on the process of MLLMs understanding documents. We compare using structured text as a reference with using only images for understanding. On MMLongBench, adding structured text information increases the accuracy of Qwen2.5-VL-7B-Instruct from 0.389 to 0.435. This 10% performance improvement fully demonstrates the importance of structure in long document reading. The structure of different elements in the evidence images is preserved as much as possible due to the instructions given to the MLLM. The structured text surprisingly enhances the ability to answer questions. On MMlongBench, every models’ performance are improved because almost all evidence pages in this dataset contain structured tables and graphs, which can be fully interpreted by the LaTeX paradigm combined with OCR text. Additionally, the accuracy of Qwen2.5-VL-7B-Instruct is significantly higher compared to Qwen2-VL-7B-Instruct, demonstrating that Qwen2.5-VL-7B-Instruct has a stronger ability to understand LaTeX and make better use of structure.

### 4.4 Attention Analysis

We follow Zhang et al. (2024b) to conduct attention analysis in order to understand why structured text matters to MLLMs. We analyze attention from two perspectives: attention distribution and attention transformation across different cases. We present the following findings to support Observation [3](https://arxiv.org/html/2506.21600v1#Thmobservation3).

#### Attention Distribution.

We consider 370 samples with the same number of image tokens to demonstrate the distribution of attention transformation with or without structured text as references when MLLMs answer questions based on evidence images. We use Qwen2.5-VL-7B-Instruct as the main model to generate answers on MMLongBench. In the case where MLLMs rely only on images, the attention distribution shows that the model is sensitive to boundary tokens of images and exhibits an uneven distribution of attention. These results indicate that MLLMs treat almost every image token equally and have no clear focus on specific regions of the original images. In contrast, [Figure 3](https://arxiv.org/html/2506.21600v1#S3.F3) shows that MLLMs are constrained when using structured text as references. MLLMs learn to focus on useful image tokens located in figures, blocks of text, or tables. This transformation of attention distribution highlights the importance of structured text.

#### Case Study.

We perform a case study to better understand attention transformation with the help of structured text. [Figure 4](https://arxiv.org/html/2506.21600v1#S3.F4) illustrates an example. The question requires MLLMs to find the percentage of West Germany respondents who view Germany’s relationship with the United States as equally important as its relationship with Russia, based on the chart and corresponding text. The attention map in [Figure 5](https://arxiv.org/html/2506.21600v1#S4.F5) shows that attention is distributed everywhere without the control of structured attention, even in the blank areas of the given image. With the constraint of structured attention, attention focuses on blocks of text, charts, and so on. We conclude that structured text helps MLLMs reduce attention loss, guides them where to look, and increases the probability of finding fine-grained evidence regions. All these constraints enable MLLMs to better understand documents and answer DocQAs more effectively.

Figure: Figure 6: Comparison of attention weights across different layers of MLLMs for images alone versus images combined with structured text.
Refer to caption: x6.png

### 4.5 Ablation Studies

We almost obtain the best results on almost every dataset and model by using structured text and images as input. We further conduct experiments on the following cases: structured text as input, OCR text as input, and the LaTeX format acting as a placeholder without specific text information. Based on these cases, we aim to demonstrate the necessity of combining structured text and images. [Table 2](https://arxiv.org/html/2506.21600v1#S4.T2) shows that cases relying only on OCR text or only on structured text result in poor performance of MLLMs. Structured text is ineffective when not input alongside images.

We conducted ablation experiments categorized by different evidence elements from MMLongBench. The results in [Table 3](https://arxiv.org/html/2506.21600v1#S4.T3) indicate that for questions in documents that necessitate answers from charts or tables, using structured input provides a more pronounced performance gain than for other types of document elements. This observation further underscores the critical role of structured input.In the cases needed images to answer questions, the accuracy is lower. This is understandable because the lack of images causes information loss in cases where MLLMs need figures from the evidence images to answer questions. The accuracy with structured text ans images is better than with OCR text and images, further highlighting the importance of structure. This experimental result convincingly demonstrates the importance of structure. We conclude that the combination of structured text and images is key to improving the ability of MLLMs in long document understanding, supporting Observation [3](https://arxiv.org/html/2506.21600v1#Thmobservation3) from an ablation perspective.

## 5 Conclusion

Preserving the structure of the input to improve the ability of general-purpose multimodal MLLMs is essential to comprehend the underlying patterns and key factors in document understanding.
This work makes a step in this line.
We propose to use LaTeX paradigm to keep the structure of OCR text, the images combined with structured text efficiently improve the accuracy of answering questions in document understanding. Furthermore, we analyze the attention transformation of different kinds of input. This study find the structured attention is the key to make MLLMs understand better in answering DocQAs after comparing other inputs.
Future work includes proposing novel and efficient structure extraction or attention control method to effectively unlock the ability of general-purpose MLLMs in document understanding.

## 6 Limitations

The structure-preserving method proposed in this paper is an early attempt to enhance the document reading capability of MLLMs.
Could more advanced and generalized structure-preserving methods be adopted, or should we design plugins from the perspective of attention in MLLMs themselves to ensure structural attention during document understanding?
We hope that future work will explore these issues and further improve document comprehension, enabling it to serve more practical applications.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan,
Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al.
2024.
Phi-3 technical report: A highly capable language model locally on
your phone.
*arXiv preprint arXiv:2404.14219*.
- Anthropic (2023)
Anthropic. 2023.
[Introducing claude](https://www.anthropic.com/index/introducing-claude).
- Bai et al. (2023)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
Lin, Chang Zhou, and Jingren Zhou. 2023.
Qwen-vl: A versatile vision-language model for understanding,
localization, text reading, and beyond.
*arXiv preprint arXiv:2308.12966*.
- Chen et al. (2023)
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and
Jiaya Jia. 2023.
Longlora: Efficient fine-tuning of long-context large language
models.
*arXiv preprint arXiv:2309.12307*.
- Chen et al. (2024a)
Zhanpeng Chen, Chengjin Xu, Yiyan Qi, and Jian Guo. 2024a.
Mllm is a strong reranker: Advancing multimodal retrieval-augmented
generation via knowledge-enhanced reranking and noise-injected training.
*arXiv preprint arXiv:2407.21439*.
- Chen et al. (2024b)
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen
Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024b.
How far are we to gpt-4v? closing the gap to commercial multimodal
models with open-source suites.
*Science China Information Sciences*, 67(12):220101.
- Cho et al. (2024)
Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2024.
M3docrag: Multi-modal retrieval is what you need for multi-page
multi-document understanding.
*arXiv preprint arXiv:2411.04952*.
- Deng et al. (2024)
Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li,
Yuan Gao, Jun Song, Bo Zheng, et al. 2024.
Longdocurl: a comprehensive multimodal long document benchmark
integrating understanding, reasoning, and locating.
*arXiv preprint arXiv:2412.18424*.
- Ding et al. (2022)
Yihao Ding, Zhe Huang, Runlin Wang, YanHang Zhang, Xianru Chen, Yuzhong Ma,
Hyunsuk Chung, and Soyeon Caren Han. 2022.
V-doc: Visual questions answers with documents.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pages 21492–21498.
- Dong et al. (2024)
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang,
Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. 2024.
Internlm-xcomposer2-4khd: A pioneering large vision-language model
handling resolutions from 336 pixels to 4k hd.
*Advances in Neural Information Processing Systems*,
37:42566–42592.
- Faysse et al. (2024)
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline
Hudelot, and Pierre Colombo. 2024.
Colpali: Efficient document retrieval with vision language models.
In *The Thirteenth International Conference on Learning
Representations*.
- Gao et al. (2023)
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin
Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023.
Retrieval-augmented generation for large language models: A survey.
*arXiv preprint arXiv:2312.10997*, 2:1.
- Han et al. (2025)
Siwei Han, Peng Xia, Ruiyi Zhang, Tong Sun, Yun Li, Hongtu Zhu, and Huaxiu Yao.
2025.
Mdocagent: A multi-modal multi-agent framework for document
understanding.
*arXiv preprint arXiv:2503.13964*.
- Hu et al. (2024)
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li,
Ji Zhang, Qin Jin, Fei Huang, et al. 2024.
mplug-docowl 1.5: Unified structure learning for ocr-free document
understanding.
*arXiv preprint arXiv:2403.12895*.
- Hui et al. (2024)
Yulong Hui, Yao Lu, and Huanchen Zhang. 2024.
Uda: A benchmark suite for retrieval augmented generation in
real-world document analysis.
*arXiv preprint arXiv:2406.15187*.
- Khattab and Zaharia (2020)
Omar Khattab and Matei Zaharia. 2020.
Colbert: Efficient and effective passage search via contextualized
late interaction over bert.
In *Proceedings of the 43rd International ACM SIGIR conference
on research and development in Information Retrieval*, pages 39–48.
- Lewis et al. (2020)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim
Rocktäschel, et al. 2020.
Retrieval-augmented generation for knowledge-intensive nlp tasks.
*Advances in neural information processing systems*,
33:9459–9474.
- Li et al. (2023)
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang,
Chunyuan Li, and Ziwei Liu. 2023.
Mimic-it: Multi-modal in-context instruction tuning.
*arXiv preprint arXiv:2306.05425*.
- Liu et al. (2024a)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a.
Improved baselines with visual instruction tuning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 26296–26306.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.
Visual instruction tuning.
*Advances in neural information processing systems*,
36:34892–34916.
- Liu et al. (2024b)
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang
Bai. 2024b.
Textmonkey: An ocr-free large multimodal model for understanding
document.
*arXiv preprint arXiv:2403.04473*.
- Lu et al. (2024)
Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, and William Yang Wang. 2024.
From text to pixel: Advancing long-context understanding in mllms.
*arXiv preprint arXiv:2405.14213*.
- Ma et al. (2024)
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan
Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, et al. 2024.
Mmlongbench-doc: Benchmarking long-context document understanding
with visualizations.
*arXiv preprint arXiv:2407.01523*.
- Mishra et al. (2019)
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.
2019.
Ocr-vqa: Visual question answering by reading text in images.
In *2019 international conference on document analysis and
recognition (ICDAR)*, pages 947–952. IEEE.
- Park et al. (2024)
Jaeyoo Park, Jin Young Choi, Jeonghyung Park, and Bohyung Han. 2024.
Hierarchical visual feature aggregation for ocr-free document
understanding.
*Advances in Neural Information Processing Systems*,
37:105972–105996.
- Suri et al. (2024)
Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan A Rossi,
and Dinesh Manocha. 2024.
Visdom: Multi-document qa with visually rich elements using
multimodal retrieval-augmented generation.
*arXiv preprint arXiv:2412.10704*.
- Team et al. (2024)
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol
Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024.
Gemini 1.5: Unlocking multimodal understanding across millions of
tokens of context.
*arXiv preprint arXiv:2403.05530*.
- Team (2025)
Qwen Team. 2025.
[Qwen2.5-vl](https://qwenlm.github.io/blog/qwen2.5-vl/).
- Tito et al. (2023)
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2023.
Hierarchical multimodal transformers for multipage docvqa.
*Pattern Recognition*, 144:109834.
- Tworkowski et al. (2023)
Szymon Tworkowski, Konrad Staniszewski, Mikołaj Pacek, Yuhuai Wu, Henryk
Michalewski, and Piotr Miłoś. 2023.
Focused transformer: Contrastive training for context scaling.
*Advances in neural information processing systems*,
36:42661–42688.
- Wang et al. (2024a)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin
Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024a.
Qwen2-vl: Enhancing vision-language model’s perception of the world
at any resolution.
*arXiv preprint arXiv:2409.12191*.
- Wang et al. (2024b)
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji,
Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. 2024b.
Cogvlm: Visual expert for pretrained language models.
*Advances in Neural Information Processing Systems*,
37:121475–121499.
- Ye et al. (2023)
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao,
Guohai Xu, Chenliang Li, Junfeng Tian, et al. 2023.
mplug-docowl: Modularized multimodal large language model for
document understanding.
*arXiv preprint arXiv:2307.02499*.
- Zhang et al. (2024a)
Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li,
Ka-Ho Chow, Conghui He, and Wentao Zhang. 2024a.
Ocr hinders rag: Evaluating the cascading impact of ocr on
retrieval-augmented generation.
*arXiv preprint arXiv:2412.02592*.
- Zhang et al. (2024b)
Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao,
Shaobo Guo, Qi She, and Shanghang Zhang. 2024b.
[cls] attention is all you need for training-free visual token
pruning: Make vlm inference faster.
*arXiv preprint arXiv:2412.01818*.
- Zhu et al. (2023)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.
Minigpt-4: Enhancing vision-language understanding with advanced
large language models.
*arXiv preprint arXiv:2304.10592*.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: see weaknesses
soundness: 2
strengths: 1. By using the latex paradigm to encode document structure, it only adjusts the input format to take effect, without modifying the model architecture or conducting additional training, resulting in low implementation costs.

2. Four benchmark datasets covering multiple scenarios such as MMLongBench are selected, and four mainstream MLLMs are used for testing, ensuring high reliability of the results.
summary: This paper focuses on the challenges of multimodal large language models in document understanding, and finds that unstructured multi-element text extracted from PDFs reduces model performance due to attention dispersion and structure loss. This paper proposes a structure-preserving method that encodes document elements using the LATEX paradigm. Through experiments on four benchmark datasets and four mainstream models, this method significantly improves the model's accuracy in document question-answering tasks.
weaknesses: 1. The method relies on the LATEX paradigm, which may have limitations in encoding complex document structures, such as irregular layouts, handwritten annotations.

2. The ablation experiments show that structured text alone performs poorly; the method must rely on image input to be effective, which limits its application in scenarios where only text data is available.

3. The text extraction process uses OCR for image-based PDFs, but this paper does not discuss how OCR errors affect the quality of structured text.

4. All experiments are conducted on 7B-scale models, but there is no exploration of whether the method's effectiveness varies with model size.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: Check Weaknesses 1-6.
soundness: 2
strengths: **1.** Clear, easy-to-reproduce idea (formatting as LaTeX) rather than architectural changes. 

**2.** Comprehensive evaluations: including multiple datasets and models evaluation (MMLongBench, LongDocURL, PaperTab, FetaTab; Qwen2-VL/2.5-VL, LLaVA-1.6-Mistral, Phi-3.5-Vision). 

**3.** Consistent experimental framing comparing images only with images+plain text and with images+structured text. This paper also includes ablations and attention visualization and analyses to support the idea.
summary: The paper propose to convert evidence pages into LaTeX “structured text” and feeding that alongside page images to improves MLLM document QA. The method is simple: prompt an MLLM to transform plain text and images into LaTeX text (tables, figures with paths, sections), then answer questions with images and this LaTeX structured text. The authors report accuracy gains on four DocQA benchmarks using several 7B-scale MLLMs compared with plain text settings and provide attention visualizations to claim “structured attention” explains the gains.
weaknesses: **1.**  Incremental novelty: converting extracted content to a markup that preserves hierarchy/layout is not new in document understanding; many systems preserve structure via HTML/Markdown/layout tokens. The specific choice of LaTeX as the markup is a thin tweak, and the paper offers little principled justification for LaTeX compared with simpler structured formats. The “Observations 1–3” are mostly empirical restatements.

**2.** Method underspecified
It’s unclear which MLLM generates the structured text, whether it is the same as (or stronger than) the QA model, and how prompts were chosen. If a stronger or differently-trained model creates LaTeX summaries, the downstream gains might is not fair. The paper does not include any the generator model of structured data or settings. 

**3.** Efficiency missing: the approach appears to expand token counts substantially, but there is no accounting of latency, context budget, or cost relative to baselines. Gains without efficiency reporting are hard to assess for practicality. 

**4.** Weak baselines: stronger structured-evidence baselines (HTML/Markdown with bounding-box, existing doc-structure tools, etc.) are missing. 

**5.** Attention analysis is over-interpreted: “structured attention is key” to a conclusion without solid evidence beyond visuals. with no rigorous causal link established.

**6.** Limited generality. In Table 3, with structured Latex data, the performance on several subsets (Generalized-text and Figure) is overall degraded. The proposed DocQA with structured Latex data does not show consistent gain for the general purpose.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: Please see weaknesses.
soundness: 1
strengths: The work is easy to follow, the datasets are standard, and the attention visualisations give a concrete illustration of why scattered tokens mislead the model. It is also useful to be reminded that “more text” is not automatically better if the layout signal is lost.
summary: The paper investigates how input format affects Multimodal Large Language Models (MLLMs) in document understanding tasks. The authors observe that plain text extracted from PDFs often degrades model performance. To address this, they propose using LaTeX-formatted structured text as input alongside images, aiming to preserve document layout and element relationships. They conduct experiments on multiple benchmarks and models (7B scale), showing that structured input improves performance and leads to more focused attention patterns. The authors claim this is a novel, training-free method to enhance document understanding without modifying model architecture.
weaknesses: 1. **Lack of Novelty in Observations:**
The central observation — that plain text extracted from PDFs can hurt performance — is not surprising and has been implicitly or explicitly noted in prior work. The degradation is largely due to loss of layout and structural context, which is well-known in OCR and document AI communities. Presenting this as a novel finding overstates the contribution.


2. **Methodological Depth: More Like Prompt Engineering:**
The proposed method — using LaTeX to represent document structure — is essentially a prompt engineering trick. While effective, it lacks technical novelty or algorithmic innovation. There is no learning, optimization, or generalization mechanism involved.


3. **Limited Model Scale and Generalization:**
All experiments are conducted on 7B-scale open-source models. There is no evaluation on larger models (e.g., 30B, 70B+) or proprietary models (e.g., GPT-5, Claude-4, Gemini-2.5)
These models are more robust to input format variations. The claimed benefits may not generalize to stronger models.


4. **Practical Applicability and Assumptions:**
The method assumes access to LaTeX-formatted representations of documents, which is rare in real-world applications. Most documents are scanned PDFs, PowerPoints, or HTML pages without LaTeX source. The current pipeline relies on MLLMs to generate LaTeX from images, which is error-prone and not scalable.


5. **Missing Baselines and Ablations:**
No comparison with other structured formats (e.g., HTML, XML, Markdown).

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. The paper provides a thorough validation of the conclusion that structured input leads to better document understanding than either pure image or unstructured text inputs. 
2. The paper is well-written, with a clear organization. The use of stepwise “observations” in the methodology section makes the reasoning process intuitive and easy to follow.
summary: This work mainly shows unstructured text extracted from PDFs can harm MLLMs' document understanding by dispersing attention and losing layout information while representing document elements in LaTeX can preserve structural relationships and focused the attention scores on correct elements. Experiments on multiple benchmarks reveal that structured inputs improve DocQA accuracy without retraining, guiding models toward semantically relevant regions and enhancing multimodal reasoning efficiency.
weaknesses: 1. The main conclusion, while empirically supported, appears too simple to sustain a top-tier conference contribution. 
2. The control of input variables is insufficiently rigorous, differences in image resolution or text formatting can cause large variations in token counts, which may affect VQA performance and should be accounted for in analysis. 
3. The attention analysis is relatively shallow. Additionally, in Figure 4, the right-side attention scores are not normalized, and the left-side comparison does not clearly convey its intended implication.


Decision: Reject