UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Authors: Jiabo Ye
Anwen Hu
Haiyang Xu
Qinghao Ye
Ming Yan
Guohai Xu
Chenliang Li
Junfeng Tian
Qi Qian
Ji Zhang
Qin Jin
Liang He
Xin Alex Lin
Fei Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: OCR-free
visually-situated language understanding
multimodal large language model
TL;DR: Low-cost instruction tuning based on the Multimodal Large Language Model for universal ocr-free visually-situated language understanding.

Abstract: Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.

UReader:
Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
Jiabo Ye Anwen Hu ..Hiyann Xu Qinghao Ye²,
Ming Yan" 21, Guohai Xu?, Chenliang Li Junfeng Tian', Qi Qian', Ji Zhang",
Qin Jin Liang He', Xin Lin', Fei Huang"
East China Normal University
DAMO Academy, Alibaba Group
Renmin University of China
jiabo. yeestu. ecnu, edu. on
[buanwen. haw, ym119608, shuofeng xhy @al ibaba-iabbii inc com
A stop sign with sticker that says says eating animals.
2020, the total value of beer imported into Canada was
Samuel Sánchez (ESP)
718,00.00
For Clarly's Sake, Please Don't Say 'Licensed under GPL, 2.....
716.55 million Canadian dollars.
Webpage
UReader
Natural Image
Document
Chart
Table
STOP AARaare
Visual Questioning:
Image Captioning: Provide a brief description of the given image.
Who was ranked between denis menchov and stephane Goubert?
Points Generation: Point several critical
Information Extraction: What is the value for wross_amounts
Text Reading:
Recognize the texts In the Image.
features this image.
Figure 1: The OCR-free Visually-situate Language Understanding performance of UReader on various types of images and tasks.
Abstract
single model achieves ofee----t-the-tttt ocr- ocr-free performance out of 10 visually-situater language understanding tasks, domains: documents. tables, charts, natural images, and webpage sereenshots. Codes and instruction-tuning datasets are released https://gihubb.co//uukccooo Younorrrweerrr/////// Young/URRadee
Text is ubiquitous in our visual world, convey- ing crucial information, such documents, websites, and everyday photographs. In this work, we propose UReader, first exploration of universal OCR-free visually-situated language understanding based on the Mul- timodal Large Language Model (MLLM), By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost much lower than previous work following domain-specific pretraining and finetuning paradigms. Con- cretely, UReader jointly finetuned on wide range of Visually-situatel Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with same format, namely text reading and key points generation tasks. We design shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our
5
Introduction
Leveraging strong Large Language Models as the language decoder, some recent works propose Mul- timodal Large Language Models (MLLMs) (Zhu et al., 2023; Liu al., 2023a; Ye et al., 2023; Li et al., 2023) and achieve promising vision-ands language understanding performance. Surprisingly. without in-domain training, these MILMS exhibit shallow zero-shot visual text recognition ability when fed low-resolution image with salient text information (Ye et al., 2023; Liu et al., 2023b). However, due to the variety of image types and the wide range of image sizes, they are still far from universal visually-situated language understanding. such as extracting information from documents, reading texts from webpuges, and visual question and answering on tables, as shown Figure
Equal contribution
Corresponding authors

---

Existing works for visually-situated language un- derstanding can be categorized into two-stage (Xu et al., 2021; ; Huang et al., 2022: Yang et al., 2021) and end-te--eeet mnd-to-end (Davis et al., 2022; Kim et al,, 2022;
performance in 8 of 10 tasks, across 5 domains.
Related Work
Lee et al., 2022) methods according to whether re- lying on an off-the-shelf OCR model or API. These works all follow a domain-speciffi pretraining and finetuning paradigm, thus leading to high training costs, e.g. end-to-end model Donut (Kim al., 2022) costs more than 192 A100 days.
Visually-situated Language Understanding aims to comprehend images containing rich text infor- mation. The image types are quite diverse, cover- ing document (Mathew et al., 2021, 2022; Stanis- lawek et al., 2021; Svetlichnayaa, 2020; Zhang al., 2023), table (Pasupat and Liang. 2015; Chen et al.,
Inspired by the shallow text recognition abil- ity of existing MILMs, in this work, we propose UReader for universal OCR-free visually-situated language understanding, which leverages the multi- modal Large Language Model via low-cost instruc- tion tuning (Dai et al., 2023). Different from previ- ous works, forgo pretraining tasks by leveraging the existing MLLM and directly finetune MLLM by taking full advantage of various Visually-situater Language Understanding datasets. To make the most of the strong language understanding ability of MLLM, we convert all tasks into the vision- language instruction tuning format. Besides, to en- hance text recognition and semantic understanding ability across diverse domains, we design auxil- iary text reading and key points generation tasks in the same instruction format. To utilize the low- resolution encoder of MLLM for processing high- resolution images and avoid blurry and distortion problems due to resizing, we propose a shape- adaptive cropping module to cut high-resolution image into multiple local images. Each image is firstly independently encoded with the frozen vi- sual encoder and trainable visual abstractor and then concatenated feed into the language de- coder. Moreover, we add learnable crop position encoding to help the model correlate local images and add a resized global image to alleviate salient information loss due to cropping.
2020), chart (Masry et al., 2022; Methani et al., 2020; Kafle et al., 2018; Kahou et al., 2018), natu- ral image (Singh et al., 2019; Mishra et al., 2019; Biten al., 2019; Hu et al., 2021), webpage screen- shot (Tanaka et al., 2021; Chen et al., 2021), etc. Tasks of Visually-situated Language Understand- ing range from visual question answering, image captioning, information extraction to natural lan- guage inference.
According to whether using off-the-shelf OCR models or APIs to recognize texts from images, existing work can be divided into two-stage models (Xu et al., 2021; Huang al., 2022; Tang et al., 2023; Yang et al., 2021) and end-to-end models (Kim et al., 2022; Davis et al., 2022; Lee et al., 2022). Two-stage work always designs pretrianing tasks to learn codss-modality alignment between visual inputs and text inputs. For example, for doc- ument understanding, UDOP (Tang et al., 2023) design a Joint Text-Layout Reconstruction task recover masked texts and layout information given the visual inputs and retained text inputs. Lay- outLMv3 (Huang et al., 2022) applics a Masked Image Modeling task to recover masked image tokens with the context of their surrounding text and image tokens. Without the help of an off-the- shelf OCR model, end-to-end models need to learn text recognition with high-resolution image en- coder during the pretraining stage. For example, Pix2Struct (Lee et al., 2022) proposes a Screenshot Parsing pretraining task, where the model needs to generate the complete HTML DOM tree with only a masked webpage screenshot as the input. Donut (Kim et al., 2022) designs a pretraining task to gen- erate all texts in the document image. These work all follow domain-speciftt pretraining and fine- tuning paradigm and therefore ask for high train- ing costs, c.g. Donut trained for more than 192 100 days. In this work, by leveraging the shal- low text recognition ability of Multimodal Large Language Models, we propose to directly perform
Our contributions in this work are four-fold:
We first propose instruction tuning with Multi- modal Large Language Models for OCR-free Visually-situated Language Understanding. We build an instraction-tuning dataset cover- ing 5 domains of visually-situated language document, table, chart, natural image, and webpuge sereenshot.
understanding:
We design a shape-adaptive cropping module to utilize the frozen low now-resolution vision coder for processing high-resolution images. OCR-free
UReader achieves

---

instruction tuning across various types of images and greatly reduce the training cost for universal visually-situtted Language Understanding.
Twitter
Large Language Model
LoRA
Multimodal Large Language Model is developed to empower the Large Language Model with multi- modality understanding ability, especially for vi- sion information. These work (Huang et al., 2023; Zhu al., 2023; Liu al., 2023a; Ye et al., 2023; Li et al., 2023; Dai et al., 2023) mainly connect a pre-trained vision encoder (usually CLIP VIT-I/14 (Radford et al., 2021)) with a strong large language model, such as LLaMA (Touvron et al., 2023). These MILMs show some emergent abilities, in- cluding shallow zero-shot text recognition ability (Liu al., 2023b). However, they are still far from universal visually-situatee language understanding. Firstly, due to the pretraining data for the vision en- coder being mostly natural images, MILMs show barely acceptable text understanding performance on natural images but bad performance on other types, such document (Liu et al., 2023b). Sec- ondly, most images for visuall-situated language understanding high-resolution. Rescaling them low resolution to adapt the vision encoder can cause the texts blurry and distorted. In this work, we propose to fully leverage the shallow text recognition ability of MLLMs and perform instruc- tion tuning to enhance its universal understanding ability across domains. Besides, we design shape-adaptive cropping module to alleviate the text blur and distortion problem.
Crop Position Encoding
Visual Abstractor
Visual Encoder
Shape-Adaptive Cropping Module
Who has more active users, Pinterest Twitter?
Figure The overall architecture of UReader.
age to H., Wo (raw resolution of the MLLM) can result in text being blurred, distorted, and unrecog- nizable. Thus we propose shape-adaptive crop- ping module. Specifically, as shown in Figure 3, we pre-define grids (g = (n, x (w)) tue S Ne, ni E N, now E N} with various shapes, where ny and nhe denote the number rows and columns of the grid and Ne denotes the maximum number of the cells (sub-images). To select a suitable grid for image I with shape H x W, two rules should be followed: (1) The grid should preserve the reso- lution of the image as much possible, and (2) the grid should fit the aspect ratio of the input image. To measure the resolution coherence and shape sim- ilarity between the image and cach grid, we calcu- late the resolution-related and resolution-agnosicc insection over union Sn and Sia as follows:
3 UReader
The primary goal of UReader is to efficiently utilize existing MILMMs for Visually-situatee Language Understanding tasks. In this work, we utilize but are not limited to, the mPLUG-Owl (Ye al., 2023) our basic MILM. Figure presents an overall architecture of UReader. The input image is firstly pre-processed by shape-adaptive cropping mod- ule (in Section 3). The resulting sub-images are then simultancously passed through the visual en- coder and visual abstractor. To enable the large language model to correlate multiple cropped sub- images, we apply crop position encoding module introduce spatial information across sub-images. (in Section 3.2).
Sn loU ((H, W), (nH now Wo))
Sm)(1g) loU ( (m W
(1)
non) (na, (w)) nw)
where loU denotes the insection over the union between two rectangles centered and aligned with each other. The matched grid is selected by maxi- mizing the matching score:
g" = arg max Sra(I,)) (m,
(2)
where g" is the selected grid. Then, we resize the input image to (n, H,, ny W.) and crop it to пy Tew local images. To maintain the global structure information of the image, we also resize the input
3.1 Shape-Adaptive Cropping Module
Images with texts have various aspect ratios and a great range of resolutions. Simply resizing the im-

---

Global
Local Images
where eijj E Di denotes the crop position embed- ding of the cell (c,, eÿ). We add the embedding the visual feature of each cell the language space via broadcasting along the dimension learnable queries: V/ = i.j + ei,j- We then reshape the vi- sual features into V E IR(N-Nx)xdd The resulting spatial-aware visual features and word embeddings of the input sentences are concatenated at sequence dimension and sent to the large language model.
Pre-defined Grids
resize
CrOP
In order enhance the language model's ability to effectively model multiple images while keeping low training costs, we freeze the origin language model and adopt the low-rank adaptation approach (LoRA) (Hu et al.. 2022).
Grid Matching
Figure 3: The Shape-Adaptive Cropping Module.
Instruction Tuning
image (HI, Wo) as global image. All images are then passed on the visual encoder and visual abstractor in parallel.
For developing universal visually-situated lan- guage understanding model that could process vari- ous types of images and perform different com- prehension tasks, we conduct low-cost instruc- tion tuning with a Multimodal Large Language Model. Without introducing any large-scale pre- training datasets, we directly ensemble multiple downstream datasets and perform joint training. Different downstream tasks are all reorganized to the unified instruction format (Dai et al., 2023). Besides, we design auxiliary text reading and key points generation tasks enhance text recognition and semantic understanding ability.
The visual encoder extracts visual feature V E (x,- from the input images I E NXXXXXxx x3 where N (ng mw) 1, H' W and dy denote the number and dimension of the extracted visual features, respectively. The visual abstractor further summarizes visual information and obtains higher semantic visual representations V' E R xN,NNx in language feature space by sev- eral learnable queries, where d, denotes the dimen- sion of language feature space and Ng denotes the number of learnable queries.
4.1 Tuning Tasks
3.2 Cropped Images Modeling with LLM
Unified downstream task. Downstream tasks of Visuall-situated Language Understanding cover Vi- sual Question Answering, Information Extraction, Natural Language Inference, and Image Caption- ing. For developing a universal model, we reor- ganize all tasks into the instruction tuning format (Dai al., 2023). Concretely, for the Visual Ques- tion Answering task, the question is directly used as the instruction: "Human: (question) } AI: (an- swer)". ". For the Information Extraction task, each category and value pair is expressed with a prompt as "Human: What the value for the (category)? ) AI: (value)". If some categories don't 't exist in the image, the value is 'None'. In the raw annotation of the Natural Language Inference task, 'I' means 'Entailed' and '0' means 'Refuted'. We reorga- nize the NLI task by constructing the instruction "Human: (statement), Yes or No? AI: (answer)", where 'Yes' means 'Entailed'. For the Image cap- tioning task, we refer to 11 prompts from LLaVa
MLLMs are mostly trained with a single image as the input. Due to the cropping module, we need to input visual features from multiple images into the language model. The 1 I-dimensional position embeddings of LLM can not reflect the spatial po- sition of each sub-image, which is critical to cor- relate local images. Therefore, we incorporate 2-dimensional crop position encoding to help the language model to understand the spatial relation- ship between cropped images. Specifically, we assign a location index (i.j) for each cell of the selected grid and obtain their row embedding and column embedding by two auxiliary embedding layers follows:
row % column
Embeddingrow (i)
Embeddingootmall (i)
riw column
ei.j
e
ei

---

(Liu et al., 2023a) instruct the model briefly describe the image and randomly choose prompt for each caption, such as "Human: Provide a brief description of the given image. AI: [caption]".
4.2 Instruction Data Resources
Document. DocVQA (Mathew et al., 2021) com- prises 50k question and answer(QA) paris on 12k document images from UCSF Industry Doc- uments Library. Infographies (InfoVQA) (Mathew et al., 2022) collects 5k diverse info- graphics from the internet and annotates 30k QA pairs. DeepForm" (Svetlichnaya, 2020) and Kleis- ter Charity (KLC) (Stanislawek et al., 2021) are two Information Extraction datasets. DeepForm contains 1.1k documents related to election spend- ing. 2.7k documents of KLC come from published reports of charity organizations.
Text Reading task. Text Recognition is basic ability for OCR-free Visuall- Language Un- derstanding. Therefore, we apply an auxiliary Text Reading task strengthen text recognition ability across different domains. With the text and position information in the image, we organize the texts in the common reading order: from top to down, from left to right. Directly utilizing all texts as targets (Kim et al., 2022) will result in the model focusing generating the starting texts and neglecting oth- ers to reduce the loss. Instead, we randomly choose a split position p from {0, L 22 5L
Table. WikiTableeQestionns (WTQ") (Pasupat and Liang, 2015) comprises 2.1k table images from Wikipedia and is annotated with 23k question and answer pairs demanding comparison and arithmetic operations. TabFact' (Chen et al., 2020) is Nat- ural Language Inference dataset, which contains 112 'entailed' or 'refuted' statements about 16k Wikipedia tables.
6* 6 ... 6 J. where L is the text sequence length. The left part is used as the input and the right one is the target. p - 0 means to generate all texts while other cases ask the model to continue reading following the input texts. Such design could enforce the model to read different parts of texts with the context. Start- ing texts always convey key information about the image, such the chart title. Therefore, we apply bigger sample rate (0.5) for the '0' position and 0. for other positions. To distinguish reading from the beginning and continuing reading, we design two groups of prompts and randomly choose 1 prompt for each sample. For example, an instruction of reading from the beginning can be "Human: Rec- ognize text in the image. AI: (all texts)" and an instruction of continuing reading can be "Human: The words on this picture are {left texts). Continue reading the text. AI: (right texts)".
Chart. ChartQA (Masry et al., 2022) collects various topics and types of charts from four sources: Statista (statista.com), The Pew research (pewresearch.orgg) OWID (ourworldindataa.og and OECD (oeed.org). It totally contains 21k chart images and 32k QA pairs.
Natural Images. TextVQA (Singh et al., 2019) filters 28k natural images with texts from Open Images V3 (Krasin et al., 2017) and annotates 45k QA pairs. To support image captioning with read- ing comprehension, TexiCaps (Sidorov et al., 2020) further collects 145k captions based on TextVQA WebPage Screenshot. VisualMRC (Tanaka et al., 2021) collects 5k full sereenshots of webpages from 35 websites. There are 30k annotated QA pairs where answers are expressed in fluent sen- tences (avg. 9.53 words) and much longer than the ones of QA datasets mentioned above.
Key Points Generation task. Large Language Models learn strong understanding ability from the tough language modeling task. Therefore, for stronger vision-and-language semantic comprehen- sion ability, we propose to design an auxiliary Key Points Generation task, which requires the model to give some key points about the image. To support this task, we collect QA pairs of each image and convert them to declarative sentences with Vicuna (Vicuna, 2023). These declarative sentences are finally regarded as key points about the image. We also build set of templates to instruct this task, such as "Human: Identify some key points in this picture. AI: (key points)".
Experiments
5.1 Implementation Details
We conduct experiments on recently proposed MLLM named mPLUG-OWl (Ye et al., 2023) with- out modifying its hyperparameters. The number of learnable queries of visual abstractor is 65. The dimension of hidden states dy and d, are 1024. For the ahape-adaptive cropping module, we set the
All templates for Text Reading and Key Points Generation tasks can be found in Appendix D.
'Superseript means the reformulated or modified version in DUE-bcnchmark (Borchmann et al., 2021)

---

Table I: Comparison with ocr-free methods on various types of visually-situated language understanding tasks. 'TSFT' means task-speific fine-tuning on the downstream dataset. 'underline" means achieving 80% SOTA perfor- mance.
Train
TS
Doc
Info
Deep Form
Visual MRC
Model
KLCC
WTQ TabFact
ChartQA
TextVQa TextCaps
Param
FT
VQA
VQA
Dessurt
127M
/
63.2
Donut
176M
<
67.5
11.6
61.6
30.0
18.8
54.6
41.8
43.5
74.4
93.91
Pix2Structease
282M
<
72.1
38.2
56.0
88
Pix2Structaagee
1.3B
76,6
40.0
58.6
95.5
UReader
86M
x 65.4
42.2
49.5
32.8
29.4
67.6
59.3
57.6
118.4
221.7
maximum number of cells No to 20 by default. During instruction tuning, the maximum sequence length is limited to 2048, and He, Wo are set to 224 match the pretrained resolution of the vi- 8.
the strong pretriaining model Pix2Structaarge in InfoVQA, ChartQA, and TextCaps. Considering that Pix2Structaage is trained more than 1 170k steps with batch size of 1024 on 128 TPUs, this val- idates that with the help of open-domain Multi- modal Large Language Models, learning costs for universal visually-situated language understanding can be greatly reduced. More detailed analysis can be found in Appendix B.
sual encoder. For LoRA, we the rank The learning rate schedule uses a linear warmup of 36 steps to le-1, followed by cosine decay to 0. The batch size is set to 256. For better conver- gence of each dataset, VQA is up-sampled times, Info VQA, WTQ, DeepForm, and KLC are up-sampled 2 times. The instruction tuning pro- cess takes 16 A100 days for 20k training steps (10 epochs).
5.4 Ablation Study
We perform comprehensive ablation experiments to validate the contribution of two auxiliary tasks, trainable architectures, cross-domain joint training and the design of shape-adaptive cropping module.
5.2 Evaluation
We use official training splits as tuning data and evaluate models on test splits. Following previous works (Borchmann et al., 2021: Lee et al., 2022), Doc and Info VQA are evaluated by ANLS (Biten al., 2019), DeepForm and KI.C eval- uated by F1 score. WTQ. TabFact and TextVQA are evaluated by accuracy. ChartQA evaluated with the relaxed accuracy (Methani et al., 2020). TextCaps and VisualMRC are measured by CIDEr (Vedantam et al., 2015). Evaluation of TexiVQA and TextCaps are performed with the official chal- lenge website.
Auxiliary Tasks. As shown Table 2, dropping the Key Points Generation task (r10 vs r2) causes a performance decrease on all domains of datasets, demonstrating that this task helps the model bet- ter understand the aition-andllnnguaa semantic. Further removing the Text Reading task (r2 vs rl) causes more significant performance degradation, which validates the importance of enhancing text recognition ability across different domains.
Trainable Architectures. Both the visual ab- stractor and LoRA in LLM are finetuned in URe- ader (r10). Freezing either the visual abstractor (r3) or LoRA (r4) causes performance decrease, which demonstrates that both the vision and lan- guage parts should be finetuned for adjusting to Visually-situated Language Understanding.
5.3 Main Results
We first compare UReader with state-of-the-art ocr- free models on 10 datasets. For fair and consis- tent comparison across all datasets, we finetune the strong and accessible baseline Dount on unreported datasets. As shown in Table 1, UReader achieves sfate-of-the-art performance in 8 tasks across 5 domains, covering Visual Question Answering, In- formation Extraction, Natural Language Inference and Image Captioning tasks. With much fewer trainable parameters (86M vs 1.3B) and without a specific finetuning stage, UReader outperforms
Cross-domain Joint Training. After removing 4 document datasets from the training data, URe- ader achieves worse performance (r10 vs r5) the table, natural image, and webpage domains, validating that images of different domains share some common characteristics and cross-domain joint training improves the universal performance. Besides, although trained without document data,

---

Table 2: Ablation study about auxiliary training tasks, trainable model architectures, cross-domine joint training and ahape-adaptive cropping. 'KPG' and 'TR' refer to Key Points Generation and Text Reading tasks, respectively. 'Abs' refers to the visual abstractor. 'Doc Data' means using 4 document datasets as training data or not. 'Global' means using a resized global image input. 'Crops' refers Ne. the maximum number of local images after cropping. 'CroPPos' refers to the crop position embedding.
Tasks KPG TR
Trainable Abs
Doe Data
Shape-adaptive Cropping Crops
Visual MRC
DocVQA WTQ
ChartQA TextVQA
LoRA
Global CropPos
rl
12
/
/
4
20
20
64.3
56.7
28.1
22.9
58.6
56.7
54.3
56.0
205.0
213.5
13
/
/
20
52.4
20.5
43.5
$4.9
194.9
14
1
/
/
/
20
59.5
23.5
58.5
53.3
177.0
15
v
d
v
V
20
46.2
27.4
59.8
54.0
185.6
22.0
13.4
24.2
34.4
157.4
17
1
1
4
58.0
24.7
58.9
55.5
215.3
18
/
1
20
64.1
27.6
60.7
56.5
210.7
19
d
d
20
62.8
26.7
58.7
55.4
181.1
r10
20
65.4
29.4
59.3
57.6
221.7
DecVOs
VisulMRC
mance (r10 19), proving the effectiveness of crop position encoding for correlating local images. For alleviating the distortion problem due resizing, we propose to crop images according to their raw aspect ratio. Figure 4 shows the frequency distribution of grids selected by our shape-adaptive cropping module VOc, Vi- SuaMMRC and WikiTTblleQusttions (the distribution on more datasets be found in the Appendix A). For aesthetic purposes, we present the distribu- tion with N₂ = Apparently, different domains of images have different shape distributions. For most document images in VQA, their height is greater than the width, while table images are the opposite. As webpages are scrollable, their screen- shots are always in the form of long rectangular shape. With the shape-adaptive cropping design, our model can easily adapt various image shapes without domain -ppecfi fine-tuning.
Figure Visualization of the frequency of selected grid with adaaiiade cropping module. The cell at row and column, denotes the selected frequency of grid (ng, nw j). Deeper colors represent higher selection frequencies.
our model achieves a 46.2 score on the DocVQA dataset, showing the potential out-of-domain un- derstanding ability of our training paradigm.
Shape-adaptive Cropping. The r6 Table represents directly tuning the mPLUG-OWl with- out any model revisions. With the shape-adaptive cropping, UReader achieves significantly better per- formance (r7 r6), showing that our cropping module is indispensable leverage pretrained low- resolution vision encoder for universal visually- situated language understanding. Besides, increas- ing the cropping numbers (r8 vs r7) improves the model's performance. Due to the resolution of each local image being constant (224x224), more crops mean higher overall resolution and therefore achieves better performance. Furthermore, adding resized global image bring a slight improvement in most datasets (r10 vs 18), validating that com- plete image could alleviate possible information loss due to image cropping. Finally, dropping crop position encoding also hurts the model's perfor-
Text distortion may pose little influence on vi- sual question answering because they are always about partial text information. But it is harmful for reading texts in the image because every text matters. For quantitative analysis of the influence of shape-adaptive design, we directly evaluate the performance of reading all texts. We choose the Bleu (Papineni et al., 2002) as the metric because directly measures the n-gram overlap between the ground-truth and predicted text sequence. The evaluation set is built by combining 100 randomly- selected test images from each dataset. As shown in Table 3, compared with cropping all images with fixed grid, UReader could better recognize texts

---

SHARE 2.5 BILLION
70 AMCIO
560 MIJUI
Tebruazy
Human: many pieces of content is shared day by Facebook users? UReoder: 2.5 BiLLION (v)
Humon:
What the first movie that won the filmlare
award for best actress? Ureader: Moggina Manasu [V]
Humon: On which date was the memorandam
Human: Who has more active users, Pinterest or Twitter? URcaddrr Twitter (v)
What the total number films with the language
Human:
February 27, 1992 (V)
15 (V)
(+)
[b]
inter,
FSF members
Wine consumption per person, 1961 to 1979
sollwar
lcending eydeens adwinotorinn, ONU Project
Human:
As an intern who will work closely with?
Human: does the read?
South Homa
UReader: BIS SIBELIUS
UReader: You will work closely
1972
Human: List handful of essential elements in this visual.
SYMPHONIES 2&5 MINNESOTA
(V)
country the highest consumption 1979 was Iceland, (V) The country with the lowest wine consumption per person in 1979 was Madagascar (x).
ORCHESTRA VANSKA COSMO (N)
(d)
(e)
(f)
Figure 5: Qualitative results of UReader. Crucial regions are enlarged for clearer visualization.
Table The Text Reading performance of UReader un- der the condition of N₂ 9. 'w/o adapt means remov- ing the shape-adaptive design and cropping the image with fixed grid x 3.
manner, when given image with rich texts, such as a page of book, the model often reads the be- ginning texts and then continues writing without watching the image. More qualitative results can be found in Appendix C. Finally, as shown in case UReader is able to list some key points about the chart by combining the title and line information. Listing key points in this work is just superficial attempt at open-ended generation, and its perfor- mance is far from promising, e.g., UReader makes mistake about the lowest line. More effort is needed towards comprehensive understanding of images with rich text.
Model
Bleu1
Bleu2
Bleu3
Bleu4
UReader w/o adapt UReader
21.4
15.4
12.0
9.7
24.9
18.1
14.3
11.7
in the image due to shape-adaptive design that alleviates the text distortion problem.
5.5 Qualitative Results
Figure 5 show some qualitative results produced by our UReader on different types of images. URe- ader could not only extract information from the document (case a), but also understand different instructions and provide corresponding answers by attending to different regions (case b). Table un- derstanding always involves layout comprehension and statistics. As shown case c, given a table im- age, UReader could well relate different columns to answer the 'first movie' and perform simple statis- ties about the 'total number'. As for images with multiple paragraphs of text, e.g. webpage screen- shot in case e, URcader could also locate the rele- vant paragraph, understand the texts and answer the question accurately. Case d shows the text reading performance. With the help of the Text Reading task, UReader is able to read texts from top left to bottom right. But, due to the language decoding
Conclusion
We first propose to leverage existing Multimodal Large Language Models for universal ocr-free visually-sitated language understanding through low-cost instruction tuning. All downstream tasks reorganized into unified instruction-tume ttuningtttnnnnnnn for- mat. Besides, we design the Text Reading task and Key Points Generation task enhance text recogni- tion and vision-and- -an-aaaggg semantic comprehen- sion abilities. To utilize the pre-trained vision en- coder for processing high-resolution images, we de- sign shape-adaptive cropping module, which cuts the image into multiple local images considering its raw aspect ratio and resolution. UReader achieve state-of-the-art ocr-free performance in 8 out of 10 datasets, ranging from documents, tables, charts, and natural images to webpage screenshots.

---

Limitations
Szyndler, and Filip Gralinski. 2021. DUE: end-to- end document understanding benchmark. In NewrlPS Datasets and Benchmarks.
Our experiments validate that UReader is able to correlate local images after cropping a high- resolution image. However, UReader struggles to understand multi-page documents (e.g. books and papers) due lacking ability correlate different pages and the limited sequence length of the de- coder. Besides, UReader feeds an equal number of features for each local image into the language decoder. But, not all local images contain rich vi- sion or text information. In the future, we will explore a more efficient way to encode different crops. Furthermore. the open-ended generation about Visually Language understanding is far from well studied. We try developing key points generation ability in this work but more difficult generation tasks are not currently considered, such as giving the chain-of-the thouth of the answer. How to simulate such abilities through instruction tuming is topic worth studying. Finally, the Text Reading task helps the model recognize texts, but the text reading performance with the LLM as the decoder is far from satisfactory due the halluci- nation problem. Instructing the LLM read texts strictly according to images is challenging topic.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact A large-scale dataset for table-based fact verification. In Inter- national Conference on Learning Representations (ICIR), Addis Ababa, Ethiopia.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji. Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. Websre: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 4173-4185.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision- language models with instruction tuning. CoRR, 0ls/2305.06600..
Brian Davis, Bryan S. Morse, Brian L. Price, Chris Tensmeyer, Curtis Wigington, and Vlad I. Morariu. 2022. End-to-end document recognition and under- standing with dessurt. In ECCV Workshops (4), vol- ume 13804 of Lecture Notes in Computer Science, pages 280-296. Springer.
Hao Feng, Zijian Wang. Ji Tang, Jinghui Lu, Wen gang Zhou, Houqiang Li, and Can Huang. 2023. Unidoc: A universal large multimodal model for simultancous text detection, recognition, spotting and understand- ing. ArXiv, abs/2308.1 1592.
Ethics Statement
Our UReader relies on multi-modal large language models that are trained on large-scale image and text data from the web and therefore may be subject to issues such as toxic language and bias (Bender et al.. 2021). However, our model is further fine- tumed on publicly available datasets and is used specifically in the domain of visually-situated lan- guage understanding, where these issues have min- imal impact.
Anwen Hu, Shizhe Chen, and Qin Jin. 2021. Question- controlled text-aware image captioning. In ACM Multimedia, pages 3097-3105. ACM.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck. Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. 2023. Language not all you need: Aligning perception with language models. CORR. abs/2302.
References
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610-623.
Yupan Huang, Tengehao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3; Pre-training for doc- ument AI with unified text and image masking. ACM Multimedia, pages 4083-4091. ACM.
Ali Furkan Biten, Rubên Tito, Andrés Mafla, Lluis Gómez Bigorda, Marçal Rusiñol, C. V. Jawa- har, Emnest Valveny, and Dimosthenis Karatzas. 2019. Scene text visual question answering. ICCV, pages 4290-4300. IEE..
Kushal Kafle, Brian Price, Scott Cohen, and Christo- pher Kanan, 2018. DVQA: understanding data visu- alizations via question answering. In CVPR, pages 5648 5656-5656. Computer Vision Foundation IEEE Computer Society.
Lukasz Borchmann, Michal Pietruszka, Tomasz Stanis- lawek, Dawid Jurkiewicz, Michal Turski, Karolina

---

Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Åkos Kádár, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. In ICLR (Workshop). OpenRe- view.net.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh. and Anirban Chakraborty. 2019. OCR- VQA: visual question answering by reading text in images. In ICDAR, pages 947-952. IEEE.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th annual meefing of the Association for Computa- tional Linguistics, pages 311-318.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Won- scok Hwang, Sangdoo Yun. Dongyoon Han, and Se- unghyun Park. 2022. Ocr-free document understand- ing transformer. In ECCV (28), volume 13688 of Lecture Notes in Computer Science, pages 498-517. Springer.
Panupong Pasupat and Percy Liang. 2015, Composi- tional semantic parsing on semi-structured tables. In ACL (1). pages 1470-1480, The Association for Com- puter Linguistics.
Ivan Krasin, Tom Duerig. Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings. Stefan Popov, Shahab Kamali, Mat- teo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. 2017. Openimages: public dataset for large-scale multi-label and multi- class image classification. Dataset available from https://storage.ee wwnngleppps.commoennnnne/////eee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πlya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 9748- PMLR.
html.
Oleksii Sidorov. Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Texteaps: A dataset for image captioning with reading comprehension. In ECCV (2), volume 12347 of Lecture Notes Com- puter Science, pages 742-758. Springer.
Kenton Lee, Mandar Joshi, Tulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandel- wal, Peter Shaw, Ming-Wei Chang. and Kristina Toutanova. 2022. Pix2struct: Screenshot parsing as pretraining for visual language understanding. CoRR, abs/2210.03347.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In CVPR, pages 8317-8326. Computer Vision Foundation / IEEE
Junnan Li, Dongxu Li. Silvio Savarese, and Steven C. Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, abs/2301.12597.
Tomasz Stanislawek, Filip Gralinski, Anna Wróblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Rosal- ska, Bartosz Topolski, and Przemyslaw Biecek. 2021. Kleister: Key information extraction datasets involv- ing long documents with complex layouts. In ICDAR (1), volume 12821 of Lecture Notes in Computer Science, pages 564-579. Springer.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. CoRR, abs/2304.08485.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, al. 2023b. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv: 07895.
S Svetlichnaya. 2020. Deepform: Understand struc- tured documents at scale.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. Visualmre: Machine reading comprehension on doc- ument images. In AAAI, pages 13878-13888. AAAI Press.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: bench- mark for question answering about charts with visual and logical reasoning. In ACL (Findings), pages 2269.2279. Association for Computational Linguis- tics.
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2023. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19254- 19264.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. 2022. Infographicvqa. In WACV, pages 2582-2591. JEBE
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawa- har. 2021. Docvqa: dataset for VQA on document images. In WACV, pages 2199-2208. IEEE.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13997
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In WACV, pages 1516-1525. IEEE.

---

Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-basee image de- scription evaluation. In CVPR, pages 4566-4575. [EEE Computer Society.
Vicuna. 2023. Vicuna: An open chatbot impress- ing gpt-4. https:/ https://githbbs com/ In-sys/ FastChat.
Yang Xu. Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei A. Florên- cio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021. Layouthmv2: Multi-modal pre- training for visually-rick document understanding. In ACI/JJCNLP (7). pages 2579-2551. Association for Computational I Linguistics.
Zhengyuan Yang. Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. TAP: text-aware pre- triningg for text-vqa and text-caption. In CVPR, pages 8751-87 1751. Computer Vision Foundation IBEE.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Yc, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi. Ji Zhang, and Fei Huang. 2023. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/2304.14 14178.
Liang Zhang, Anwen Hu, Jing Zhang, Shuo Hu, and Qin Jin. 2023. MPMQA: multimodal question answering on product manuals. CoRR, ths/2304.09660)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4; Enhancing
Vasion-language understanding with advanced large language models.

---

A Grid Distribution Downstream Datasets
B.2 Compared with Pipeline Methods
We list the performance of statc-of-the-arr pipeline models in Table 4. We can summarize from the results that there are two distinct aspects. Firstly, our model achieves comparable or slightly worse results compared to the pipeline methods on TextVQA, CharlQA, InfoVQA, TextCaps and Tab- Fact. Secondly, there is a obvious gap between our model and pipeline methods on Voc,QA, Deep- Form, KLC. WTQ and VisualMRC.
We visualize the frequency distribution of grids selected by our shape-adaptive cropping module on all ten datasets in Figure 6. The wide variety of image shapes in downstream tasks highlights the crucial role of the shape-adaptive cropping module.
B Detailed Analysis on Performance
B.1 Underperforms Ocr-Free Baselines on VQA and DeepForm
For the first aspect, there are two reasons for the similarity performance: (1) Modeling the di- verse relationship between visual objects and text presents challenges for both pipeline- -based meth- ods and OCR-free methods. TextVQA, TextCaps and Info VQA requires the relation understanding between text and visual objects (i.e. logos, icons and common objects). ChartQA asks for trend com- prehension of lines. Understanding such complex cross- modalityyyy relation is challenging for both ocr- free and pipeline methods. (2) The simplicity of task formats can reduces performance gaps. Tab- fact is simply binary classification task resulting the small performance gap.
It can be seen that UReaderunderperforms ocr-free baselines on Doc and DeepForm. There are two main factors: (1) Donut performs the pre- training on large-scale document dataset IIT-CDIP (1 IM document images), which is the same domain VoA and DeepForm. But UReader does have a pretraining process and is just instruction finetuned on ensembled datasets (less than 0.5M assorted images). Training with more document images brings better performance. (2) The pretrain- ing task of Pix2struct is to predict the HTML dom tree of masked web screenshot, which requires the model to fully understand the layout informa- tion of the image. But UReader is trained to read texts from top down, from left right, which requires a weaker layout understanding ability. The pretraining on layout understanding also leads to improved performance on VQA.
For this second aspect, the main performance gup appears in three categories of datasets: docu- ment, table, and webpage screenshot. The reasons are two folds: (1) The gap in terms of text recogni- tion and layout extraction. In document, table and website, text is the dominant information source and the layout(e.g. row and column layout in table) is relatively uniformer than the chart and natural images. Therefore, with pre-extracted texts and layout information, the image. But for OCR-Free models, such as our UReader and Donut, it's still challenging to fully recognize all texts. (2) The gap in terms of mod- eling capacity on multi-page document input. for multiple- page document datasets KLC (98% 4 pages) and DeepForm (75% pages), OCR-Free models only input the first page and lose much information.
The conclusion can also be substantiated by the observations on the other two datasets (i.c., In- fo VQA and KLC) included in the document do- main as previous work (Tang al., 2023). For the InfoVQA dataset, the image is poster style and the layout not as important as VQA and Deep- Form but the relationship between text and vision objects matters more, like natural image and chart image. As for the KLC dataset, ocr-free models are only fed with the first page (always the cover of report). where the layout is much simpler than DocVQA and DeepForm. Therefore, URead- ercan outperform baselines on these two document datasets.
more easy to understand
B.3 Zero-shot Performance
In summary, compared with ocr-free model Donut and Pix2Struct, due to the pretrianing of MLMM on open-domain datasets, UReaderis bet- ter at understanding cross-modality relationships in the image but weaker at comprehending text layout information without large-scale document pretraining and specific layout understanding tasks.
We test the zero-shot performance of UReader on unseen dataset OCR-VQA. With the same evalua- tion metries, UReader outperforms mPLUG-OWl (41.1 vs 28.6) and a recent work UniDoc (Feng et al., 2023) (41.1 vs 34.5) with the training of lay- out prediction. The results show that the zero-shot performance of our method on unseen domains is

---

CA
Te
CopsCops
WileTahlloQustion
TabTact
MRC
For
InfagraphcaVOOA
DocVOA
Kleisterhhrrit
Figure Visualization of the frequency of selected grid with the shape-adaptive cropping module on downstream datasets.
DocVQA
InfeVQAA
DaepFiorm
WTQ
TlaFFict
CharQA
TeaviQA
TeuCaps
VisuaMMCC:
OCR-Pipline:
84 TUDDP)
85.5 S5.SUDDP))
47.2(U1)XP))
72, (UUDP)
70 ((De))))))
56 NPh-STU)
139 (PeSTTty
364 2i ayyuiTT
UReader
65.4
42.2
49.5
32.8
29,4
118.4
221.7
Table 4: Performance comparison between UReaderand state-of-the-art pipeline methods.
acceptable.
these images. The original mPLUG-OWl is used for comparison.
More Qualitative Results
In Figure 12 (a), UReader is able to accurately recognize and answer questions about the small text in natural images ("Name of passenger" and "MORRIS/KARLA")) In contrast, mPLUG-OWl does not respond with the name in the first round and gives an incorrect answer even with prompt in the second round.
C.1 Downstream Results
More qualitative results natural images, charts, tables. documents and webpage screenshots are shown in Figure 7-11.
Figure 11 show a sample of Text Reading and Visual Question Answering about webpage screenshot from VisualMRC. As mentioned in Sec- tion 5.5, when given instruction about reading all texts in the image, UReader can read the be- ginning texts but sometimes is easy to continue to generate vision-irrelovant texts. With appropriate instructions, UReader could indeed recognize texts in other regions, such as 'exercise increases cellular recycling'. Therefore, the hallucination problem during text reading is not because UReader can- not recognize texts, but the generating manner of LLM decoder. When beginning texts are read from the image, the decoder may generate the following texts according to the closer text context rather than the image.
In Figure 12 (b), we raise query consisting of two cascaded questions, which requires the model to simultancously understand the spatial position of the non-textual objects referred to by the query and locate the corresponding fields. It can be seen that the UReader completes this task well, while mPLUG-Owl answers incorrectly in both object recognition and price extraction.
In Figure 12 (c), conduct multi-turn conver- sions with UReader included references to the history of the conversa- tion. Some questions also require a certain amount of common sense. For example, the time of account creation is equivalent to the time of joining Twitter, and the inactive state of the Follow button indicates that the user has not followed Iron Musk. UReader answers these questions well. mPLUG-OWl can correctly recognize the Iron Mush, but is it prone to generating content that unrelated to the image.
screenshot. The questions
C.2 Open-domain Results
We present open-domain examples in Figure 12. We use randomly collected images and freely ask questions to the model based on the content of

---

Human: Recognize text in the picture. UReader: WILLIAM G. MATHER
Human: Is this ship used for a business? UReader: yes
Human: Who's name the ship? UReader: william g. mather
Fit
Human: Provide brief description of the given image.
UReader: A large ship named William G. Mathers is docked pier.
(a)
World graphics Day 2010
Human: What words are in the image?
UReader: graphics World Day 2010 #wgd2010
Human: What day is it?
UReader: world graphics day
Human: What year shown? UReader: 2010
Human: Render clear and concisc summary of the photo.
UReader: A ycllow and black pencil sits wooden table with the date 2010 on it.
Regg
Figure 7: Text Reading, Visual Question Answering and Image Captioning performance of UReader on natural images from TextVQA. Correct and wrong answers are colored green and red, respectively.
Global slews of Trump's cheracteristics
Human: List the words in the image.
ciborry Aldd Prwilier Thwsrrr
UReader: Global views Trump's churacteristics who say they think of
Aecder
President Donald Trump as. Arrogant 75 Intolerant 65 Dangerous 62 A strong
leader 55 Charismatic Well-qualified to be president 26 Caring about ordinary people 23.
Cangrou
Bock
Human: What percent who think of President Donald Trump as Dangerous?
Owrkmatk
UReader: 62.
Human: the of the percents of Charismatic and Well-qualified
president than A strong leader?
hoo now vass good Dow. on
UReader: Yes.
(a)
Human: What dose the picture say?
prinunt FORTY rith INL 100H
UReader: Final to primary energy ratio (%), 1990 2010 Albania 80.0 60.0 Sudan 40.0 20.0 0.0 1990.0 1995.0 2000.0 2005.0 2010.0
80%
Bonin
Human: Which country represents in red line?
UReader: Benin.
60%
Suaar
Human:
Which country data below 80% for the last five years?
UReader: Sudan.
(b)
Figure 8: Text Reading and Visual Question Answering performance of UReader charts from ChartQA. Correct and wrong answers are colored green and red, respectively.
leading to erroneous statements.
the prices for each component, notice that it proactively filters the components with price of $0, making the answer more consistent with the user's intention. It indicates that UReader can find the form related to the question and compre-
In Figure 12 (d), we ask the UReader about the price and its components based on an image consist- ing of multiple forms. Although UReader wrongly includes the header in the answer and does not list

---

Human: Highlight few significunt elements this photo. UReader: The constructor that had the most points Ferrari. constructor had the most retirements was Williams- McLaren. The constructor that the most laps completed was Ferrari. The constructor that had the most laps completed was Ferrari. The constructor that had the most laps completed was Ferrari.
Ruburs lorihars
del
Human: How long did it take michael schumacher to finish the race?
UReader: 1:34:45.026
8m
(a)
chart postiers
Human: What was edelweiss's biggest hit?
UReader: "Bring Me Edelweiss"
1981 53
1884
Tehing Estlwess
No. Lennueee"
Sayes
1992
how many years after can't get no.. Edelweiss" did the single "bring me edelweiss" (remix) appear?
Human:
UReader:
(b)
Figure 9: Key Points Generation and Visual Question Answering performance of UReader on tables from Wik- iTableQuestions. Correct, wrong and repetitive answers are colored green, red and gray, respectively.
DATE: Jarch 22, 1052
Human: Identify some key points this picture.
UReader: The document is dated March 22, 1991. The document is from the United States.
A. C. Monk
Human: What
of the Dealer UReader: C. Monk
DEALER
How many grass/ /trss/straw pieces of matter found in the core samples ? UReader: 2
Human:
EYEE
GERAR/ELLRY
Mint/Srcine
22
Human:
How many lint'string pieces of matter is found in the core samples? ? UReader: 2
(a)
A BRIEF HISTORY OF SOCIAL MEDIA PLATFORMS
Human: Parse text from imagery. UReader: A BRIEF HISTORY OF SOCIAL MEDIA
PLATFORMS 1978 The first public Bulletin Board System went online. 1979 UseNet created by Duke University students Tom
Truscott and Jim Ellis. 2. 1982 Six Degrees, the first social media website, is launched. 2002 Linkedln is created by Reid Hoffman. 2003 MySpace launched several eUniverse employees. 2004 Mark Zuckerberg officially launches Twitter. 2006 Twitter is launched by Jack Donsey used by employees ut Oden. 2007 2008 The photo-sharing app Instagram launched. Tumbir is launched by David Karp. 2010 The blogging site Tumblr is launched by David Karp. 9. CREATED BY: Carly Racklin / Ghost Tweeting ghostweeting
2010
The photo shartngapp
Inttagramis bunched.
Human: Which social media site launched in 2010? UReader: Instagram.
ghoattwesting
(b)
Figure 10: Text Reading, Key Points Generation and Visual Question Answering performance of UReader on documents from Doc VQA and InfoVQA. Correct and wrong answers are colored green and red, respectively.
hensively understand the meaning of each field in the form. In contrast, mPLUG-Owl generates re-
sponses that are full of illusions due to the loss of textual information in the image.

---

ARTICLER WITH THE TOMC AUTOPMAST
Human: Detect words in the photogruph:
UReuder: ARTICLES WITH THE TOPIC: AUTISM
Could molecular handcuffs lower the protein that causes autism? By Timothy D. Rayne May 20, 2012 Huntington's discase ncurodogenerative disorder that causes progressive decline in cognitive unction and motor control. It is caused by mutation gene that codes for protein called buntingtin.....
cockdre handouff lover the protels that CRUIRE
What suggested that molecular bandcuff's could lower?
Human:
UReader:
The protein that Huntington's disease
Exercise increases cellular recycling
Human: What does exercise increase?
UReader: increases cellular recycling.
Lolp seeiliiing
Figure 11: Text Reading and Visual Question Answering performance of UReader webpage screenshot from VisualMRC. Correct and wrong answers are colored green and red, respectively.
These results reveal that UReader maintains some interactive ability of MLMM in the open domain and shows stronger visually-sitateed lan- guage understanding ability.
D Instruction Templates
The instruction templates of the auxiliary Text Reading and Key Points Generation tasks are shown in Table 5.

---

MES
LEMONS
APPLES
$0,35
BANANA
Watermaoon
Apricots
45e
MORR KARLA
UReader
UReader
Human: What is the fruit the top left and what is the price of it?
Human: Who is the passenger?
UReadde:: moris/karia
URender; melon, $1.90
mPLUG-Owl
mPLUG-OW]
Human: Who is the passenger?
Human: What is the fruit the top left and what is the price of it?
mPLUG-Owl: The passenger in the image woman.
Human: What's ber name?
mPLUG-Owl: The fruit on the top left is lemon, and the price $1.
mPLUG-Owl: The name of the passenger is "Mary."
(a) Text understanding in natural image secnario
(b) Spatial-aware comprehension involving non-textual objects.
websenibes
Sanva Mapa
Diael
USColars
Elon Murk
TT messe 18 DOSARE DIE www.
UReader
Human: Who is he and how many people he is following
UReader: Elon Musk, 334
UReader
Human: When he create his twitter account?
Human: How moch should pay?
UReader:
UReader: $136.38
Human: Have I followed him?
Human: What are the components of the payment?
UReader: No
UReader: Service Name, Amazon CloudFront, AWS Data Trunsfer Service, Amazon Simple Storage Service, Amazon
mPLUG-Owl
Simple DB, Amazon Elastic Compute Cloud
Human: Who is he and how many people he is following mPLUG-OWf: The man in the image Elon Musk, famous entrepreneur, engineer, and inventor, He following 10 people on Twitter, including himself.
UP..UG-Ow!
Human:
How much should pay?
mPL UG-Owl: Based on the image, total amount you should pay for the Amazon Web Services (AWS) bill is $1 $1,000.
Human: When did he create his twitter account? mPLUG-Owf: Elln Musk created his Twitter account in June
2006, making him one of the earliest adopters of the social media platform.
Human: What ure the components of the puyment? mPLU PPLU--O::: The payment coasists cleck and credit card. The check is for SI, $1,000, and the credit card for
Human: Have I followed him?
mPLUG-Owf: Yes, you have followed Elon Musk on Twitter.
$1,000.
(c) Multi-tum information extruction
(d) Multi-tum form understanding
Figure 12: Comparsion with mPLUG-Owl on open-domain sistaaly-situated Language Understanding. The key words in the answers and the key regions in the images are annotated with the same color. The incorrect response of UReader colored red.

---

Table 5: Instructuion templates used for text reading from the beginning. continue reading and key points generation tasks. The complete instruction for continuing reading is random combination of a prompt from part A and another one from part B.
Task
Part Instruction Template
<lmage>Human: what words are in the image? AI: (all texts).
<lmage>Human; what texts are in the picture? AI: [all texts).
<lmage>Human: what does the image read? AI: (all texts).
<lmage>Human,
Human; what does the picture say? AI: (all texts).
<Image>HHuman what is written in the image? AI: (all texts).
<lmage>Human; list the words
iunage. AI: (all texts).
<Image>Human: list the
the picture. AE: (all texts].
<Image>Human; Recognize text in the image. AE: [all texts].
text reading from the beginning
<lmage>Human: Identify text in the picture. Al: alll texts].
<lmage>Humant Deciphering written content in the photo. (all texts].
Iumge>lluman: Extract words from the graphic
[all texts].
<Image>Human
Human: Parse text from imagcry. Al: (all texts).
<lmage>Human: Read written language
visuals. AI: texts).
IImage>Human Decode from the snapshot. Al: (all texts).
dumaa= Translate text in the picture. AI: [all texts].
<Image>Humaa Retricve written information from the image. Al: [all texts].
<Image>Humant Detect words in the photograph. Al: (all texts].
Amaage>Human: The picture reads [left texts).
<lmage>Human: The image says [left texts).
<lmage>Human: There words [left texts] the image.
<lmage>Human: Words [left texts) are in the picture.
<lmage>Hluman: The texts in this image read leff texts]. <lmage>1lumant
words on picture are [left texts].
A
Almage>Iluman: The script depicted in this image reads [left texts).
Humage>Iluman: The writing on this visual representation states (left texts).
continue reading
elmage>HHuman Human: The content presented in this diagram states (left texts).
<Image>Human: The language used in this photograph says [left texts).
<Image>Human; The inscription
this picture explain [Jeft texts).
Continue reading the text. AI: |right texts).
Read following text. Al: (right texts].
Read the text behind. Al: rright texts].
What is the following text? Al: (right texts].
<umage>Human: Identify some key points this picture. Al: (key points]. <laaee>Iumann Point out several critical features in this image. Al: [key points).
<lmage>1 luman: Highlight few significant elements in this photo. Al: (key points].
<lmage>I luman: Give
essential details
illustration.
[key points).
<lmage>HHuman: Draw atlention to some important aspects in this diagram. Al: (key points).
key points generation
<lmage>Human: Mention a couple of crucial points in this snapshot. Al: (key points].
<lmage>Human: Indicate a few pertinent items this graphic. Al: (key points).
<lmage>Human: Outline some significant characteristics in this image. Al: (key points).
<lmage>Human; Specify some key components in this picture. Al: (key points).
<lmage>HHuma; List handful of essential elements in this visual. Al: (key points).


Reviews:
[{"review_id": "jpoazjiPlY", "reviewer": "Reviewer_v2FU", "summary": "", "questions": "- Given that UReader performs well in 8 out of 10 OCR-free tasks,  I am wondering what are reasons for not achieving leader scores in the other two tasks (DocVQA, DeepForm). What are some specific challenges encountered in these two tasks and what are some plans for improvements?\n\n- Could you elaborate on the particular areas or factors where UReader falls short when compared to OCR pipeline models? What are the components that the authors thin UReader is missing to achieve human-level understanding?\n\n-  Table 1: Shouldn't the model be considered as finetuned if the training data is included in the evaluation? What is the performance like if the model is then further finetuned on the evaluated dataset?\n\n- The methodology for the text reading task involves randomly selecting a split position $p$ to process the input text in a conversational format. Could you provide some insights into the importance of this processing step? How would the model's performance be affected if all texts were treated solely as targets (i.e., $p=0$)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1 (Reject 1): Novelty and Relation with mPLUG-Owl.\n\nUReader is proposed to extend a MLMM to universal visually-situated language understanding. In this work, we utilize but not limited to mPLUG-Owl. Rather than naively training a MLMM on a combined instruction-formatted dataset, the novelty of our model design lies in mainly two aspects: \n\n1. In terms of model structure,  rather than training a high-resolution image encoder like Donut or Pix2Struct, we propose a shape-adaptive cropping module to fully leverage the original low-resolution (224x224) image encoder for processing high-resolution document images (e.g. 896x1120). As mentioned in Lines 481-488, this cropping module is critical for the universal visually-situated language understanding performance, e.g., dropping this cropping module (table 2, r6 vs r10) causes -43.4 on DocVQA, -35.1 on ChartQA, -64.3 on VisualMRC.\n2. Besids unifying different tasks of different datasets, we further design Text Reading and Key Points Generation tasks to strengthen models' text recognition and semantic understanding abilities. As validated in our ablation study, these two auxiliary tasks significantly improve the performance, e.g., +8.7 on DocVQA, +6.5 on ChartQA, and +16.7 on VisualMRC.\n\n### Q2 (Question 1): Dataset performance variation, challenges and improvement plans.\n\nUReader underperforms ocr-free baselines on DocVQA and DeepForm because of two main factors:\n\n1. Donut performs the pretraining on large-scale document dataset IIT-CDIP (11M document images), which is the same domain as DocVQA and DeepForm. But UReader does no have a  pretraining process and is just instruction finetuned on ensembled datasets (less than 0.5M assorted images). Training with more document images brings better performance.\n2. The pretraining task of Pix2struct is to predict the HTML dom tree of a masked web screenshot, which requires the model to fully understand the layout information of the image. But UReader is  trained to read texts from top to down, from left to right, which requires a weaker layout understanding ability. The pretraining on layout understanding also leads to improved performance on DocVQA.\n\nThe conclusion can also be substantiated by the observations on the other two datasets (i.e., InfoVQA and KLC) included in the document domain as previous work [1]. For the InfoVQA dataset, the image is poster style and the layout is not as important as DocVQA and DeepForm but the relationship between text and vision objects matters more, like natural image and chart image. As for the KLC dataset, ocr-free models are only fed with the first page (always the cover of a report) , where the layout is much simpler than DocVQA and DeepForm. Therefore, UReader can outperform baselines on these two document datasets.\n\nIn summary, compared with ocr-free model Donut and Pix2Struct, due to the pretrianing of MLMM on open-domain datasets, UReader is better at understanding cross-modality relationships in the image but weaker at comprehending text layout information without large-scale document pretraining and specific layout understanding tasks.\n\nIn the future, we will study how to add as less extra document images as possible and design an auxiliary layout prediction task to strengthen the layout comprehension ability.\n\n[1] Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., ... & Bansal, M. (2023). Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19254-19264).\n\n### Q3 (Reject 2, Question 2): Compare with existing OCR-pipeline models. And the missing for human-level understanding.\n\nThanks for your advice about comparing our model with SOTA pipelines for more in-depth analysis. We list the performance of SOTA pipeline models below. \n\n| |DocVQA|InfoVQA|DeepForm|KLC|WTQ|TabFact|ChartQA|TextVQA|TextCaps|VisualMRC|\n|-|-|-|-|-|-|-|-|-|-|-|\n|OCR-Pipline|84.7(UDOP)|47.4(UDOP)|85.5(UDOP)|82.8(UDOP)|47.2(UDOP)|72.9(UDOP)|70.5(DePlot)|56.3(PreSTU)|139.1 (PreSTU)|364.2(LayoutT5)|\n|UReader|\t65.4|\t42.2|\t49.5|\t32.8|\t29.4|\t67.6|\t59.3|\t57.6|\t118.4|221.7|\n\nWe can summarize from the results in the table that there are two distinct aspects. Firstly, our model achieves **comparable or slightly worse results** compared to the pipeline methods on **TextVQA, ChartQA, InfoVQA, TextCaps and TabFact**. Secondly, there is a **obvious gap** between our model and pipeline methods on **DocVQA, DeepForm, KLC, WTQ and VisualMRC**.\n\nFor the first aspect, there are two reasons for the similarity performance:\n\n1. **Modeling the diverse relationship between visual objects and text presents challenges for both pipeline-based methods and OCR-free methods**. TextVQA, TextCaps and InfoVQA requires the relation understanding between text and visual objects (i.e. logos, icons and common objects). ChartQA asks for trend comprehension of lines. Understanding such complex cross-modality relation is challenging for both ocr-free and pipeline methods. \n2. **The simplicity of task formats can reduces performance gaps**. Tabfact is a simply binary classification task resulting the small performance gap.\n\nFor this second aspect, the main performance gap appears in three categories of datasets: document, table, and webpage screenshot. The reasons are two folds:\n\n1. **The gap in terms of text recognition and layout extraction**. In document, table and website, text is the dominant information source and the layout(e.g. row and column layout in table) is relatively uniformer than the chart and natural images. Therefore, with pre-extracted texts and layout information, it is more easy to understand the image. But for OCR-Free models, such as our UReader and Donut, it’s still challenging to fully recognize all texts. \n2. **The gap in terms of modeling capacity on multi-page document input**. for multiple-page document datasets KLC (98% > 4 pages) and DeepForm (75% > 1 pages), OCR-Free models only input the first page and lose much information.\n\nTaking into account both two factors, UReader is not competenat on document, table and website domains. \n\nWe will add this comparison and analysis in the revised version, In the future, to narrow this gap, we will further improve our model, including how to process multiple pages with a memory bank and how to strengthen the model’s layout comprehension with an auxiliary task. Besides, visually situated language understanding always requires math computation (e.g. table and chart understanding), to achieve human-level understanding, strengthening the math ability or incorporating a math component is necessary.\n\n### Q4 (Reject 3): Relatively low performance on unseen data domains, e.g. documents, as demonstrated in Table 2 (r5 vs r10).\n\nImpressive zero-shot performance on unseen data domains is a challenging problem for not only OCR-free models but also pipeline models. SOTA pipeline models [1,2,3,5] all follow a pretrain and finetune framework. This is because the layout style and importance of different modalities vary a lot in different image types. For example, the row and column layout is crucial for table understanding, the vision semantic is important for poster-style documents, and the line trend is critical for chart understanding.\n\nRecently, a pipeline work [4] tried utilziing LLM for zero-shot document image understanding by designing layout-aware prompts. With the Alpaca (7B) as the LLM and carefully designed prompt, its pipeline zero-shot performance on DocVQA is 42.0, while ours ocr-free zero-shot performance is 46.2 (Table 2, r5). Besides, we further test the zero-shot performance of UReader on unseen dataset OCR-VQA, with the same evaluation metrics, UReader outperforms mPLUG-Owl (41.1 vs 28.6) and a recent (Aug 19) work UniDoc [6] (41.1 vs 34.5) with the training of layout prediction.\n\nThis shows that under our framework, the zero-shot performance on unseen data domains is acceptable. We agree that such zero-shot performance is far from promising and will further study how to improve its zero-shot abilities in the future, such as introducing layout prediction tasks to strengthen the general vision understanding abilities.\n\n[1] Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022. ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 3744–3756.\n\n[2] Chenliang Li, Bin Bi, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021. StructuralLM: Structural Pre-training for Form Understanding. In ACL 2021.\n\n[3] Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking. In ACM MM 2022. arXiv:2204.08387\n\n[4] Wang, Wenjin, et al. Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering. arXiv preprint arXiv:2306.00526 (2023).\n\n[5] Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., ... & Bansal, M. (2023). Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19254-19264).\n\n[6] Feng, H., Wang, Z., Tang, J., Lu, J., Zhou, W., Li, H., & Huang, C. (2023). UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding. arXiv preprint arXiv:2308.11592.\n\n### Q5 (Question 3): Table 1 should not use FT. And if the model is then further finetuned on the evaluated dataset?\n\nThanks for your advice, we will more clearly mention that the \"FT\" flag represents specific finetuning on the downstream datasets to avoid misunderstanding. As you suggested, we further independently finetune UReader on 5 datasets, including DocVQA, WikiTableQuestion, ChartQA, TextVQA and VisualMRC, covering 5 categories. As shown below, the 1st row is the raw performance of the unified UReader. The following rows illustrate the performance of the model after continuous fine-tuning on each dataset.\n\n| |Document (DocVQA)|Chart (ChartQA)|Table (WikiTableQuestion)|Natural (TextVQA)|WebPage (VisualMRC)|\n|-|-|-|-|-|-|\nInstruction-tuned|65.4|59.3|29.4|57.6|221.7|\nContinuous fine-tuned on DocVQA (5.0k steps)|**66.5**|57.2|27.1|53.3|219.1|\nContinuous fine-tuned on ChartQA (2.0k steps)|58.1|**61.9**|24.5|49.1|194.8|\nContinuous fine-tuned on WikiTableQuestion (1.0k steps)|63.2|59.1|**29.7**|54.1|217.7|\nContinuous fine-tuned on TextVQA (1.1k steps)|58.6|57.0|26.3|**57.9**|202.4|\nContinuous fine-tuned on VisualMRC (1.8k steps)|63.54|59.25|27.63|54.76|**228.5**|\n\nWith further finetuning, there are indeed performance increases on corresponding datasets (e.g. +6.8 on VisualMRC, +2.6 on ChartQA, + 1.1 on DocVQA). However, this can cause performance decreases on other datasets. By utilizing existing MLLM, we aim to develop a single and unified model that could handle as many types of images as possible. Therefore, taking into account the balance of performance and versatile ability, we think the UReader with only unified instruction tuning is better. We will add this analysis to the revised version.\n\n### Q6 (Question 4): Insights into the importance of this processing step (text reading task involves randomly selecting a split position).\n\nAs discussed in lines 315-318, we have noticed that the model's ability to predict text based on longer contexts is weaker compared to shorter contexts. During the inference process, the model often overlooks visual information and produces words inconsistent to the image when generating longer sequences. Therefore, we increase the training ratio for predicting text with longer contexts by randomly selecting starting points. \n\nFor instance, when employing the teaching-forcing method on the sequence $T=[t_0,t_1,t_2,t_3,t_4]$, we calculate the loss value for predictions $P(t_i|ctx_{<i})\\, i \\in [1,2,3,4]$. If we specifically choose i=3 as the starting point, we build an extra training example and compute the loss value for predictions $P(t_i|ctx_{<i})\\, i \\in [3,4]$. As a result, loss values are calculated twice for $t_3$ and $t_4$ in this scenario.\n\nThis approach can aid the model in gaining a more comprehensive understanding of the image.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Achieves the state of the art results in 8 out of 10 tasks among OCR-free models with efficient training.\n   - This is due to authors' optimization of training techniques, such as visual instruction tuning, LoRA, that enable fast training with small number of parameters to tune (16 A100 days with 86M) compared to prior work, e.g. 1.3B Pix2struct model that requires a batch size of 1024 on 128 TPUs.    \n  - Model training is reproducible compared to such large models proposed in prior work.\n- Dense ablation studies in Table 2 show that each contribution of auxiliary tasks such as text reading (TR) and keypoints generation (KPG), and components in shape-adaptive cropping module bring improvement in performance.\n- Qualitative examples are impressive in their fine-grained text recognition skills despite not relying on OCR. \n- Model figures are clear to understand for future implementations and how the input images are processed for text recognition abilities.", "reasons_to_reject": "- Work could be seen as limited novelty as the proposed UReader can be perceived primarily as an extension of the mPLUg-Owl model trained on instruction-formatted OCR dataset.\n\n- While the proposed model demonstrates a solid performance against other OCR-free models, the authors do not thoroughly compare these results with those of existing OCR-pipeline models. It would be good to know how the model performs in the broader landscape of visually situated language understanding models. \n\n- Relatively low performance on unseen data domains, e.g. documents, as demonstrated in Table 2 (r5 vs r10). This means that the model needs to be trained on the image domains to perform well, which is the major weakness of OCR free v.s. pipeline model that is not as affected by the image modalities.", "questions_to_authors": "- Given that UReader performs well in 8 out of 10 OCR-free tasks,  I am wondering what are reasons for not achieving leader scores in the other two tasks (DocVQA, DeepForm). What are some specific challenges encountered in these two tasks and what are some plans for improvements?\n\n- Could you elaborate on the particular areas or factors where UReader falls short when compared to OCR pipeline models? What are the components that the authors thin UReader is missing to achieve human-level understanding?\n\n-  Table 1: Shouldn't the model be considered as finetuned if the training data is included in the evaluation? What is the performance like if the model is then further finetuned on the evaluated dataset?\n\n- The methodology for the text reading task involves randomly selecting a split position $p$ to process the input text in a conversational format. Could you provide some insights into the importance of this processing step? How would the model's performance be affected if all texts were treated solely as targets (i.e., $p=0$)?", "paper_topic_and_main_contributions": "The paper develops UReader, a vision language model developed for OCR-free visually situated language understanding.  It leverages an existing visual instruction tuning model (mPLUg-OWL) in their collection of unified instruction dataset covering general downstream task, text reading task, and key points generation. To deal with dense coverage of texts in the images, the authors propose a new shape-adpative cropping module that can easily adapt to various image shapes without domain-specific fine-tuning. Experiment results show that the model achieves state of the art ocr-free performance in 8 out of 10 visually situated language understanding task without finetuning. Ablation studies show interesting empirical findings: 1) auxiliary tasks of text reading and key points generation improve the model performance, 2) vision and language parts should be finetuned to adjust to their tasks, 3) shape adaptive cropping module with more number of crops, encoding cropped positional encoding, and adding resized global image  all bring improvements in performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L164, 170: \"visuall\" -> visually \n- While the comparison to OCR free VL models is reasonable one, it would be helpful for readers to know the performance of models that use an OCR system to understand the gap between OCR free and reliant system. \n- Table 2 can be broken down to multiple tables to clearly see the effects of the auxiliary tasks and cropping module. It is difficult to immediately understand the high-level story looking at this Table."}}, {"review_id": "RdkqDvp4WM", "reviewer": "Reviewer_4gdp", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Limited contribution of the paper.\n\nThanks for your appreciation  for  the design of the shape-adaptive cropping module for processing high-resolution images.  For  achieving universal visually-situated language understanding, it's necessary to tackle various sizes of images. Existing MLLM could only process 224x224 or 448x448 images, which can not recognize texts in high-resolution document or website images. Therefore, how to leverage existing vision encoder process high-resolution images is the most difficult problem for developing a universal MLLM with low training cost. As one of the contributions in our work, the shape-adaptive cropping module is an elegant solution that avoids re-training a high-resolution encoder and could be adapted to other MLMM. \n\nBesides this model deisgn, there are another two contribution in our work:\n\n1. As for the framework, we **first propose to develop universal visually-situated language understanding by only performing instruction tuning and dropping the high-cost domain-specific pretraining** used in previous methods like Donut and Pix2Struct. Under this framework, our method achieves OCR-free SOTA performance on 8/10 tasks and greatly reduces the training cost. Our model is trained on 20k steps and costs 16 A100 days, while Donut costs 192 A100 days and Pix2Struct-large is trained on 170k steps on 128 TPUS. \n2. Besids unifying different tasks of different datasets, **we further design Text Reading and Key Points Generation tasks to strengthen models' text recognition and semantic understanding abilities**. As validated in our ablation study, these two auxiliary tasks significantly improve the performance, **e.g., +8.7 on DocVQA, +6.5 on ChartQA, and +16.7 on VisualMRC**.\n\nConsidering these  aspects,  we argue that our contribution is three-fold and not just a multi-task finetuning.\n\n### Q2: The usage of FT flag in Table 1. \n\nThe \"FT\" flag represents independent finetuning on the specific downstream datasets. We will more clearly mention that the \"FT\" flag to avoid misunderstanding. Different from such a setting,  UReader is instruction-tuned with an ensembled dataset and trained with different tasks. With further specific finetune, UReader could also perform better on corresponding datasets. As shown in below, we further independently finetune UReader on 5 datasets, including DocVQA, WikiTableQuestion, ChartQA, TextVQA and VisualMRC, covering 5 categories. The 1st row is the raw performance of the unified UReader, The following rows illustrate the performance of the model after continuous fine-tuning on each dataset.  With further finetuning, there are indeed performance increases on corresponding datasets (e.g. +6.8 on VisualMRC, +2.6 on ChartQA, + 1.1 on DocVQA). However, this can cause performance decreases on other datasets. By utilizing existing MLLM, we aim to develop a single and unified model that could handle as many types of images as possible. Therefore, taking into account the balance of performance and versatile ability, we think the UReader with only unified instruction tuning is better. We will add this analysis to the revised version. \n\n### Q3: The discussion about the model size of UReader.\n\nTrainable parameters listed in the table 1 aim to validate that to achieve universal visually-situated language understanding, leveraging an MLMM with our framework spends much less training cost than previous ocr-free models following the domain-specific pretrianing and finetuning framework. When comparing the whole model size,  more comprehensive abilities should be considered. For example, as shown in the FIgure 12 in the appendix, our single model presents multi-turn information extraction and form understanding abilities given diverse human instructions, which is not feasible for Donut or  Pix2Struct. Besides, due to that raw parameters of the LLM is frozen, our model is still able to process pure-text instruction as LLaMA, which is not also feasible for Donut or  Pix2Struct. We will explain more about the trainable parameters in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper achieves promising experimental results and the method to process high-resolution images is interesting.", "reasons_to_reject": "1. My main concern is the limited contribution of the paper. UReader can be seen as a multi-task fine-tuning on a wide range of visually-situated language understanding datasets such as DocVQA and InfographicsVQA. The more interesting part for me is the method to process high-resolution images.\n2. In Table 1, UReader also performs fine-tuning (multi-task fine-tuning) on downstream tasks. \n3. The train parameters of UReader are much smaller than other methods. But UReader uses CLIP-Large and LLaMA-7B as initialization, the total parameters of UReader are much larger than Pix2Struct.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents UReader, which is built upon a multimodal large language model and is capable of OCR ability.\nUReader is tuned on a wide range of visually-situated language understanding datasets including documents, tables, charts, natural images, and webpage screenshots.\nThe paper also designs a shape-adaptive cropping module to utilize the frozen low-resolution vision encoder for processing high-resolution images.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UjjP1ON2br", "reviewer": "Reviewer_4B6Y", "summary": "", "questions": "Please check the \"Reasons To Reject\" section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Model's zero-shot abilities on unseen datasets and unseen tasks.\n\nAs mentioned In lines 470-480, we remove 4 document datasets from the training set and perform **zero-shot testing on DocVQA dataset** (table 2, r5).  Compared with a recent work utilizing LLM for zero-shot document image understanding by carefully designed layout-aware prompts [1], our zero-shot performance is much better **(46.2 vs 42.0)**.  Besides, we further test the **zero-shot performance of UReader on unseen dataset OCR-VQA**. With the same evaluation metrics, UReader outperforms mPLUG-Owl **(41.1 vs 28.6)** and a recent (Aug 19) work UniDoc [2] **(41.1 vs 34.5)** with the training of layout prediction. These two experiments show that **the zero-shot performance of our method on unseen domains is acceptable**.\n\nBesides, as shown in the figure 12, our model presents **multi-turn information extraction and multi-turn form understanding abilities**, which are unseen task formats during instruction finetuning. Therefore, our model shows some **generalization capabilities with unified instruction tuning**. We will further study how to strengthen this ability, such as developing chain-of-the-thought abilities in different domains.  Multi-turn information extraction.\n\n[1] Wang, Wenjin, et al. Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering. arXiv preprint arXiv:2306.00526 (2023).\n\n[2] Feng, H., Wang, Z., Tang, J., Lu, J., Zhou, W., Li, H., & Huang, C. (2023). UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding. arXiv preprint arXiv:2308.11592.\n\n### Q2:  Explain \"although trained without document data\" on line 477. The performance analyses of DocVQA.\n\nAs mentioned In lines 470-472, The r5 in table 2 represents removing 4 document datasets from the training data. Line 477-480 refer to the  r5 rather than the UReader, so this is consistent. \n\nUReader underperforms ocr-free baselines on DocVQA because of two main factors:\n\n1. Donut performs the pretraining on large-scale document dataset IIT-CDIP (11M document images), which is the same domain as DocVQA. But UReader is just instruction finetuned on ensembled datasets (less than 0.5M assorted images). Training with more document images brings better performance.\n2. The pretraining task of Pix2Struct is to predict the HTML dom tree of a masked web screenshot, which requires the model to fully understand the layout information of the image. The pretraining on layout understanding leads to improved performance on DocVQA.\n\nThe conclusion can also be substantiated by the observations on the other two datasets (i.e., InfoVQA and KLC) included in the document domain as previous work [1]. For the InfoVQA dataset, the image is poster style and the layout is not as important as DocVQA but the relationship between text and vision objects matters more, like natural image and chart image. As for the KLC dataset, ocr-free models are only fed with the first page (always the cover of a report) , where the layout is much simpler than DocVQA and DeepForm. Therefore, UReader can outperform baselines on these two document datasets.\n\nIn summary, compared with ocr-free model Donut and  Pix2Struct, due to the pretrianing of MLMM on open-domain datasets, UReader is better at understanding cross-modality relationships in the image but weaker at comprehending text layout information without large-scale document pretraining and specific layout understanding tasks.  \n\nIn the future, we will study how to add as less extra document images as possible and design an auxiliary layout prediction task to strengthen the layout comprehension ability.\n\n[1] Tang, Z., Yang, Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., ... & Bansal, M. (2023). Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19254-19264).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Utilizing instruction tuning for OCR-free Visually-situated Language Understanding tasks makes sense\n\n2. The performance is good\n\n3. Ablation studies are exhaustive, showing the effects of different modules\n\n4. The paper is well structured", "reasons_to_reject": "1. If I understand correctly, the model is instruction tuned and evaluated on the same set of datasets. However, I believe instruction tuning can also help models to generalize to unseen datasets/tasks. I think it would be valuable to evaluate the model's zero-shot abilities on unseen datasets and unseen tasks, which would largely increase the impact of this work.\n\n2. Lack of analyses of the performance on DocVQA. Also, DocVQA is one of the instruction tuning dataset, while authors said \"although trained without document data\" on line 477, is this inconsistent?", "questions_to_authors": "Please check the \"Reasons To Reject\" section.", "paper_topic_and_main_contributions": "This paper instruction tunes a pre-trained Multi-modal Large Language Model (MLLM) for OCR-free Visually-situated Language Understanding tasks. It converts an array of datasets into the instruction tuning format, tasks covering document, table, chart, natural images, webpage. Furthermore, the authors propose a Shape-Adaptive Cropping Module to better understand images with various resolutions, and Cropped Images Modeling for enhancing spatial relationship understanding. The final results show the effectiveness of this approach.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new approach to multimodal NLU using multimodal large language models. Two reviewers see a lot of potential in the paper, one is more reserved, but their reviews are less substantiated. The authors addressed several of reviewers’ concerns.
Decision: Accept-Findings