Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Authors: Kenton Lee
Mandar Joshi
Iulia Raluca Turc
Hexiang Hu
Fangyu Liu
Julian Martin Eisenschlos
Urvashi Khandelwal
Peter Shaw
Ming-Wei Chang
Kristina Toutanova
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised
pretraining
screenshots
parsing
language
vision
transformers
interfaces
charts
figures
tables
documents

Abstract: Visually-situated language is ubiquitous---sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain-specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy, we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve state-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images. 

Under review as a conference paper at ICLR 2023
PIX2STRUCT: SCREENSHOT PARSING AS PRETRAIN- ING FOR VISUAL LANGUAGE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
Visually-situated language is ubiquitous sources range from textbooks with di- agrams to web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to this diversity, previous work has typically relied on domain- specific recipes with limited sharing of the underlying data, model architectures, and objectives. We present Pix2Struct, a pretrained image-to-text model for purely visual language understanding, which can be finetuned on tasks contain- ing visually-situated situated language. Pi Pix2Struct is pretrained by learning to parse masked screenshots of web pages into simplified HTML. The web, with its rich- ness of visual elements cleanly reflected in the HTML structure, provides large source of pretraining data well suited to the diversity of downstream tasks. In- muitively, this objective subsumes common pretraining signals such as OCR, lan- guage modeling, image captioning. In addition to the novel pretraining strategy. we introduce a variable-resolution input representation and a more flexible inte- gration of language and vision inputs, where language prompts such questions are rendered directly on top of the input image. For the first time, we show that a single pretrained model can achieve sfate-of-the-art results in six out of nine tasks across four domains: documents, illustrations, user interfaces, and natural images.
INTRODUCTION
Research on the interaction between language and vision has traditionally focused on tasks where images and text can be separated into distinct channels, e.g. visual question answering or image captioning. However, visually-situated language is far more pervasive way in which these modal- ities interact and blend together. For example, documents, tables, infographics, and user interfaces (UIs) are intended to be consumed holistically, without clear boundaries between textual and visual elements (Figure MD. Comprehensive understanding of this information requires deep set of skills, including the ability to recognize text, understand language, and incorporate diverse visual context.
Previous work understanding visually-situadl on complex task-specific combinations of available inputs and tools. For example, document- understanding models (Huang et al., 2022) rely on external OCR systems, Ul-understanding models rely on platform-specific structural metadata (e.g. Android view hierarchy) (Bai et al.] (021), and inngraa-uundertanddin
language is scattered. The focus is typically
models rely on diagram parses (Kembhavi al. 2016). Domain-speent speccifc en- gineering can be effective for high-resource settings such as documents, where there is an abundance of tools and data available. However, these pipelined models lack sharing of the underlying data, model architectures, and objectives across domains, limiting their general applicability. Moreover, relying on external systems like OCR increases engineering complexity, limits adaptability, and can increase overall computational cost. Recent work on OCR-free, end-to-end document understanding from images (Kim et al. 2022 Davis et al. 2022) has attempted to remove such task-specific engi- neering and reliance external components during inference by learning to decode OCR outputs during pretraining significant step towards more general-purpos models. However, the focus on just text at the surface level limits the depth of knowledge transferred from unsupervised data. Effective use of pixel-only models remains an open challenge.
We present Pix2Strued inputs with the generality and scalability provided by self-supervieed pretraining from diverse and abundant web data. Specifically, we propose screenshot parsing objective that requires predicting
pretrained model that combines the simplicity of purely pixel-level

---

Under review as a conference paper at ICLR 2023
Sereenshot Parsing Pretraining
Hkun coamcll
Wee ai Redtvet jer coneersed
Chanceler?
MINE
Pricing
dusily but stfccie priing you conerta buttoees the vr' sanorssso.
Free
50/mo
$15
$29/mo
<<Fro>
carnivore
lis: of videos for weather reporte different locatiors
Fred LeCrone
<<<515> </mo>>
useza iacluded>
<10 CB of storage>
<Friority emeil pupport>
accass>>
started>s>
Figure 1: Examples of visually-situated language understanding tasks, including diagram QA (AI2D), app captioning (Screen2Worde) and document QA (DocVQA). We also include an exam- ple of our proposed pretraining task (screenshot parsing) on the left. Pix2St ruct directly encodes the pixels from the input image (above) and decodes the output text (below).
an HTML-based parse from masked screenshot of a web page. HTML provides clean, vital signals about text, images, and layouts, while the masked inputs encourage joint reasoning about their occurrence. With the diversity and complexity of textual and visual elements found on the web, P ruxt learns rich representations of the underlying structure of web pages, which we show can effectively transfer to a variety of downstream visual language understanding tasks.
A key ingredient which enables this transfer is processing inputs visually and holistically they are intended for human readers. We introduce variable-resolution inputs for vision transformers that prevent distortion of the original aspect ratio, which can vary greatly across documents, figures, and Uls. During finetuning, we render other inputs (e.g., questions in VQA and bounding boxes in UI tasks) onto the image input for the task. In effect, we consume all our inputs though a single modality, simplifying the modality combination problem in previous work.
We train two variants with 282M and 1.3B parameters, which we refer to Pix2Struct-BB and P i 1222truuttt Luct-Lagee respectively, on 80M screenshots of web pages from the C4 corpus (Rafiel al. 2020). Experiments on four domains and nine tasks show that our finetuned models strongly outperform Donut (ranging from 9 to 53 points), the strongest existing baseline without pipelines. Compared with baselines with domain-specific pipelines, we lag behind the state of the art high- resource domains such as documents and natural images, but we observe significant improvements (ranging from to 44 points) in low-resource domains such as illustrations and Uls. We hope that these results encourage the community continue developing such general-purpose methods and further enable new applications in this currently fragmented intersection language and vision.
To summarize, major contributions
follows:
We introduce the area of general-purpond visually-situater language understanding, which con- sists of diverse tasks but common challenges.
We propose screenshot parsing pretraining objective based on the HTML source of web pages. We show that our objective more effective than previous attempts at enabling the elegant pixel-to-text design for general-purpose visually-situated language understanding.
We introduce criiable-reolutionn input representations the Vision Transformer and new fine- tuning strategies that seamlessly integrate language and vision inputs by directly rendering any language prompts top of the input image.
The pretrained checkpoints and code for reproducing results for all nine tasks available at https://qithube com/ httpc///nooommm/c///p///ttttttt.
2

---

Under review as a conference paper at ICLR 2023
2 METHOD
2.1 BACKGROUND
Prior attempts at pixel-only modeling of visually situated language have largely focused on docu- ments and natural images. For documents, Donut (Kim et al. (022) and Dessurt (Davis et al. 2022) combine pretrained objectives based on surface-level features from synthetic images or predicted OCR outputs. For natural images, concurrent work-GITH (Wang et al., (022a) and PaLI (Chen et al. 2 focus on collecting and training on large scale image captioning data that transfers well to datasets with natural images (e.g. TextCaps).
We aim to provide a single pretrained model that can be finetuned on a wider variety of tasks and domains. The input to model is an image in the form of raw pixels only, and the output of our model is text in the form of token sequences, similar to Donut. The goal is visual analog of models like T5 (Raffel al. 2020), where the generality of simple inputs and outputs combined with the power of pretraining on large unsupervised sources of data. During finetuning, the complexity of adapting diverse downstream tasks resides only in data preprocessing.
Even without visual context, pixel-only language modeling for text has only recently been at- tempted (Rust al., 2022) ppehhaps because it requires solving multiple hard sub- aoob-rrolems. First, the ability to read with high fidelity while at the same time building rich high-level representations poses difficult optimization problem. Second, encoding text-heavy inputs (e.g. long documents) involves processing high-resolution images with variable aspect ratios. State-of-the-art document understanding models (Huang et al., 2022) therefore rely on the combination of (possibly noisy) OCR outputs with low resolution images. We argue that the reliance OCR has prevented explo- ration of learning more general- pennrra--urppoe purpose representations that meaningfully extend beyond the text.
We show the various components of Pix2Struce that address these challenges. Section 2.2 discusses modifications to the transformer inputs to handle variable aspect ratios and resolutions. We then discuss our proposed screenshot parsing objective (Section 2.3) and how curriculum learning leads to more robust transfer learning (Section (2.4). Finally, Section [2.5] shows how Pix2Struct consumes textual and visual inputs (e.g. questions and images) in the same space by rendering text inputs onto images during finetuning.
2.2 ARCHITECCURE
Pix2Struct is an image-nncoder--ex--eeoode based on the Vision Transformer (ViT) (Dosovit- skiy et al. 2)2T). While the bulk of the model is fairly standard, we propose one small but impactful change to the input representation to make P pix2st ruct more robust to various forms of visually- situated language. Before extracting fixed-size patches, the standard ViT scales the input images to a predefined resolution, which creates two undesirable effects: (1) rescaling the image distorts the true aspect ratio, which can be highly variable for documents, mobile Uls, and figures. (2) transferring these models to downstream tasks with higher resolution is non-trivial (Touvron et al. 2019 Wang jet al. 2021b), since the model only observes one specific resolution during pretraining.
We instead propose to always scale our input image up or down such that we extract the maximal number of patches that within the given sequence length (see Figure Appendix [B). In order for the model handle variable resolutions unambiguously, embeddings for the input patches. Together these changes to the standard ViT inputs provide two major ad advantages in terms of robustness to: (1) extreme aspect ratios, which is common in the domains that we experiment with, and (2) on-the-fly changes to the sequence length and resolution.
use 2-dimensional absolute positional
2.3 PRETRAINING
The goal of pretraining is for Pix2Struce to represent the underlying structure of the input image. To that end, create self-supervised pairs of input images and target text from web corpus. For each web page in the pretraining corpus, we start by collecting its screenshot and HTML source.
Screenshot parsing inputs & outputs The screenshot and HTML modified to ensure rich and dense learning signal during pretraining. These modifications provide a reasonable trade-off between preserving the semantics of the page and requiring practical decoder sequence length.
3

---

Under review as a conference paper at ICLR 2023
<<<Python>
Programming Survey
Programming Survey
cimg_ars-py_logo.o ing_oltmpythone <<C+>>
Your Dooore progneenin lagagge
You Dooote progaexiing lagguge
Pytoe
Ling_ero=cpp_logo.o Img_all=C+19>
C-ra
Cenaa Regi
<<Jave>
Lang_etsrjawv_logppppp img smg_altmJava>> <Suamit>>>
Subre
Figure 2: Toy illustration of input-output pairs (right) sampled from the original web page (left).
We condense the HTML DOM tree by (I) only keeping nodes with visible elements or descendants with visible elements and (2) if node does not contain visible elements and it only has a single child, replacing the singleton child with any grandchildren to remove chained nesting. In each node, we only keep information about text and images, as represented by their filename and any alternative text. Much more information could be retained, such as element tags, style, bounding boxes, titles and URLs etc in future work. The decoder sequence length is further reduced by finding the largest subtree that fits within predefined sequence length when linearized. A bounding box indicating the region covered by the chosen subtree is also drawn on the screenshot.
For better context modeling, introduce BART-like (Lewis et
2020) learning signal by mask- ing 50% of the text while decoding the entire subtree. The masked regions are randomly sampled spans of text from the chosen subtree where we draw crossed-out opaque bounding boxes (Figure
Comparison to existing pretraining strategies Our proposed screenshot parsing seamlessly inte- grates signals reminiscent of several well-known pretraining strategies:
Recovering the unmasked parts of the parse is similar to OCR, a prerequisite skill for under- standing language. OCR pretraining was proposed in Donut which uses synthetic renderings or predicted OCR outputs. In Figure 2 predicting <C++> exemplifies this learning signal.
Recovering the masked parts of the parse is much like masked language modeling (Devlin et al, 2019). A major difference is that the visual context often provides additional sharpening the predictions. In Figure 2
useful for predicting <Python> exemplifies this learning signal.
Recovering the alt-text from images is a common pretraining strategy for image caption- ing (Sharma et al.) 20188; Waan et al. [022a]] Chen et al. (0022b). A major difference is that the model is permitted to use the web page as additional context. In Figure 2 predict- ing img.alt=C++ exemplifies this learning signal.
Appendix
Appendix[Fcconainn Fontains more examples of screenshots paired with their gold and predicted parses.
2.4 WARMING UP WITH A READING CURRICULUM
While we can directly pretrain Pix2struct on the screenshot parsing task, we find that doing this naively can result in instability and slow learning. However, if we first expose the model to a short, intense "warmup" stage of simply learning to read, we find a strong curriculum learning effect where (1) pretraining more stable and converges faster, and (2) we observe better finetuning performance, as discussed in Section Specifically, we create images of text snippets with random colors and fonts on white background. The model simply needs to decode the original text (see Appendix for an example). This type of curriculum learning was also used in Dessurt (Davis et al. 2022) and can also be viewed simplified version of Donut's pretraining.
2.5 FINETUNING
Finetuning P Pix2Struce is straightforward and largely matter of preprocessing the downstream data to unambiguously reflect the task in the image inputs and text outputs, analogous to the way T5 (Rffel al. 2020 is used for text-based tasks. In this section, cover the preprocessing strategies for the tasks described in Table Examples this preprocessing are shown in Figure I
Captioning is the most straightforward, since the input image and the output text can be directly used (as in TextCaps, Screen2Wordsy In the case where the focus of the caption is a specific bounding box (as in Widget Captioning), we draw the target bounding box the image itself.
4

---

Under review as a conference paper at ICLR 2023
For visual question answering (as in OCR-VQA. ChartQA, DOc Infographics while multimodal models typically reserve a specialized text channel for the question, we opt to instead directly render the question header at the top of the original image. P Pix2Struct reads both the question and the image jointly via the visual modality. This strategy is analogous to the common practice of simply concatenating all inputs during finetuning of pretrained text models, first proposed in GPT (Radford et al. 2018) and has been the default method in NLP since then. Intuitively, this strategy effective because Pix2St interactions between various parts of the input image. In the case of multiple choice answers (as in AI2D), we also render the choices in the header as part of the question.
has been pretrained to be sensitive to long-range
The most complex scenario is RefExp, where the task is choosing between UI components that natural language expression could be referring to. For each candidate, we create training instance where the input image contains the bounding box and referring expression, and the decoding target is "true" or "false". We sample five negative candidates per positive candidate during training. During inference, we pick the candidate for which the model generates "true" with the highest score [
EXPERIMENTAL SETUP
3.1 BENCIIMARKS
We evaluate rix2Stzae Struuttttttt on multiple benchmarks centered around visually-situated language un- derstanding across four domains: illustrations, user interfaces, natural images, and documents. Since we are the first to aggregate datasets with this scope, we optimized for diversity in domains as well as in fask-format. Evaluation is restricted to standard splits without additional labeled data. Table provides a summary of the datasets with details discussed in Section 4
We use evaluation metrics as defined in the original papers: (a) average normalized Levenshtein similarity (ANLS) for Doc and Infographic (b) exact match (EM) for A12D, RefExp, and OCR-VQA, (c) relaxed accuracy (RA) for ChartQA, and (d) CIDEr for the generation tasks.
3.2 IMPLEMENTATION AND BASELINES
Pretraining We pretrain two model variants: (a) base model with 282M parameters including 12 transformer layers with a hidden size of 768, and (b) a large model with 1.3B parameters including 18 layers with hidden size of 1536. Both models have the same warmup stage using text rendered from BooksCorpus (Zhu et al. 2015) lasting 30K steps with a maximum input sequence length of 128 patches. The base model then pretrained further for 270K steps with the screenshot parsing objective using a batch size of 3072 on 64 Google Cloud TPUs. The large model is pretrained for 170K steps with batch size of 1024 on 128 Google Cloud TPUs. Both models use an input sequence length of 2048 patches and are optimized using Adafactor (Shazeer & Stern 2018). The learning rate schedule uses linear warmup of 1000 steps 0.01, followed by cosine decay to 0. The decoder sequence length is 128 tokens, and we choose pretraining targets to have at most 1024 characters. As reference point, the base model reaches 30 BLEU and the large model reaches 32 BLEU. Details about finetuning can be found in Appendix D
Baselines Across all tasks, we found an exceedingly large number of methods which could serve as basclines. We compare our results against state of the art (SotA) methods in each domain (sec Section for method descriptions). Several methods use model ensembles, multitask with labeled training data from other datasets (Powalski et al. 2021 Wang et al.) 2022a), or validation data for training (Li et al. 2021a). For fair comparison and ease of experimentation, we focus on single- model and single-task baselines trained on standard splits. Several (per-task) SotA (Li et al. [2021c] Masry et al., (202)) use domain-specific inputs (e.g. view hierarchies for UIs or gold data tables for charts) making difficult to apply them to other domains. For a strong, consistent visual baseline across domains, we finetuned Donut on all tasks where a purely visual baseline was unavailable
'or the lowest score if something other than "true" was generated (almost always "false".) Exceptt RefExp due to the complexity of required inference modifications.
5

---

Under review as a conference paper at ICLR 2023
Chart
OCR
Ref
Widget
Screen2
Text
Doc
Info
Method
Pretraining
A12D
QA
VQA
Exp
Cap
Words
Caps
VQA
VQA
State of the w/ pipelines
(VTP)
(DQAN)
(LAT))
(UIB
(VUT)
(VUT)
(PuL
(LLM3))
(TSDDU)
45.5
38.5
67.5
90.8
97.0
64.3
160.4
83.4
46.1
GIT2
Image captioning
70.3
145.0
Pixel only
Donut
OCR
41.8
30.8
66.0
127.4
56.4
74.4
67.5
11.6
Pix2Struct
Base
Screenshot parsing
56.0
40.9
69.4
92.2
133.1
107.0
88.0
72.1
38.2
Large
Screenshot parsing
58.6
42.1
71.3
94.2
136.7
109.4
95.5
76.6
40.0
Table I: Pix2struct outperforms prior visual methods on 8 out of 9 benchmarks with SotA pretraining understandably helps on TextCaps,
results 6. While GIT2's image captioning- our screenshot parsing objective transfers to a wider variety of downstream tasks. The individual pipeline SotA methods are described in Section 4 with full results in Appendix C
RESULTS
We discuss here the main results comparing Pix2struct with prior work, as shown in Table D
4.1 ILLUSTRATIONS
ChartQA (Masry et al. 2022 is a VQA dataset with questions based on charts, i.e. visual repre- sentations of tabular data VisionTTaPas (Masry et al. 2022), the current SotA, is pipeline which operates on data tables predicted from the given charts. It consists of (1) a vision transformer en- coder for encoding the chart image, (2) TaPas encoder for encoding the question and the data table, and (3) cross moosl encoder. In contrast, P 3ix2Struce does not rely on noisy table extractors and uses the given chart directly- immpetly-imrrovin the SotA from 45.5 to 58.6 with the large variant.
A12D (Kembhavi et al. 2016) contains multiple choice questions based on illustrative science dia- grams (about geological processes, biological structures etc.). The dataset comes with only train and test splits. We set aside 1% of the train split for validation. The current SotA DQA-NET (Kembhavi) (ct al., (016)) focuses on modeling entity relationships via pipeline of tools for extracting arrows, blobs, and other visual elements. P Pix2Struct-Land
outperforms DQA-NET and Donut by 3.6 and 11.27 points respectively without any domain -specffc modifications.
OCR-VQA (Mishra et al. 2019) is VQA dataset images of book covers. The questions based on book metadata such as title, author, genre etc. Much of work on -VQ, including the pipeline SotA LATr (Biten et al. 2022), uses off-the-shelf OCR. Concureet work, GIT2 (Wang et al. 2022a), the current SotA, is pretrained on 12.9B image caption pairs. Their final finetuning stage is preceded by intermediate finetuning 2017), VizWiz-VQA (Chen et al., 2022a), and OCR-VQA (Mishra et al., (019)) amongst others. Despite not using more labeled training data, we outperform GIT2 by almost I point.
eight VQA datasets including VQAv2 (Goyal al.,
4.2 UIs
RefExp (Bai al. 2021) Given natural language referring expression, an app screenshot, and set of components (via bounding boxes on the screenshot), the goal here is to retrieve the component that the expression refers to. UlBert (Bai et al. 202I), the current SotA. is pretrained on combi- nation of inputs from mobile apps including screenshots, OCR text, and Android view hierarchies. Our models substantially ourperform UI Bert by 1.4 and 3.4% absolute, with P x2St -Large setting the new state of the art with 94% accuracy.
Widget Captioning (Li et al. 2020b) is an image captioning task where the input is an app screen- shot annotated with a single bounding box denoting widget (e.g. button or scroll bar). The caption describes the functionality of the widget (c.g. find location). VUT [i et al. 2021c), the cur- rent SotA uses a specialized UI encoder combining images, bounding boxes, and view hierarchies. P xix2Structt-lan improves the state of the art CIDEr from 97.0 to 136.7.
We evaluate on the task without the gold data table

---

Under review as a conference paper at ICLR 2023
Soreen2Words (Wang al. 2021a) is image captioning task where the imput is app screenshot and the caption describes the functionality the page (see Figure for an example). Pix2Struct-Large improves the state of the art CIDEr from 64.3 to 109.4.
4.3 NATURAL IMAGES
TextCaps Concurrent with our work, GIT2 (5.1B parameters) and PaLI (17B parameters) have ad- vanced the state of the art on TextCaps by pretraining on 10B+ image-caption pairs extracted from the web. PaLI (CIDEr 135.4 without OCR) and GIT2 (CIDEr 145) show comparable performance when finetumed without OCR based imputs. PaLI achieves SotA (CIDEr 160.4) performance when finetuned with OCR, indicating that even large-scale methods, end-to-end pixel-only performance lags behind pipeline SotA. While their image captioning-based pretraining understandably helps on TextCaps, previous work (Kim et al., 2022) shows that captioning does not necessarily transfer to other domains like documents. Moreover, screenshot parsing subsumes signals from image caption- ing (Section 2.3 while using a fraction of the the data used for pretraining GIT2 and PaLL. Overall, these results indicate that Pix2Struct could benefit from scaling even further in future work.
4.4 DOCUMENTS
DocVQA (Mathew al. 2021 is a dataset of questions about scanned documents include type- written, printed, handwritten and born-digital text. Pix2Struct-Large outperforms Donut, the previous visual SotA on Doc by points. Top-performing single-task methods like Lay- outLMv3 (Huang et al. (022) (ANLS 83.4) typically use three components: (a) an off-the-shelf OCR system, (b) pretraieed text and image encoders, and (c) additional pretraining on the IIT-CDIP scanned documents corpus. Despite using purely visual representations and no in-domain pretrain- ing data, P ruct achieves competitive performance (ANLSS 76.6).
InfographicVQA (Mathew et al. 2022) is a dataset of questions about infographics from the web. A unique challenge this dataset is its large images with extreme aspect ratios. Donut scales images to fixed aspect ratio, which we speculate is the cause of its poor performance with an ANLS of 11.6.
sets the state of the art amongst visual models with an ANLS of 40.
For both Doc and InfographicVQA. text-only baselines are at near the state of the art. A T5-based model (TS 2D + U) with 2D positional biases (Borchmann et al.) 2021) achieves ANLS of 81 on VQA and SotA ANLS of 46.1 on Infographic VQA. This is part due to the text- heavy nature of the data (especially VQA) where visual context plays a lesser role, and the more mature pretrained text-based encoders do the heavy lifting.
Common trends Overall, Pix2struct outperforms Donut in all tasks underscoring the effec- tiveness of our pretraining. We also advance the single-task state of the art on six out of nine bench- marks across four domains. Scaling up from base to large results in considerable improvements on all tasks despite the base model making 4.5 times as many iterations over the data compared the large version. Results indicate that further scaling up of wx22Strrcttt is promising direction.
5 ANALYSIS
Ablating pretraining objectives Table analyzes the importance of each component of our pre- training recipe on DocVQA, Widget Captioning, and TextCaps validation sets. The full pretraining recipe consists of a warmup reading stage on the books corpus followed by pretraining using the screenshot parsing objective. For these experiments, we use the base variant with a total of 100K steps of pretraining including 30K warmup steps followed by 70K steps screenshot parsing. The screenshot parsing ablation removes the screenshot parsing stage altogether and uses extended warmup stage of 100K steps. The warmup ablation removes the warmup stage and pretrains the next stage (from random initialization) for 100K steps. The masking ablation uses 30K steps warmup (like the full model) followed by 70K steps of screenshot parsing without masking 5
ffoo the UCSF Industry Documents I. Library https: /www. i industrydocuuuentttt www.ttt..........c..oooo... 'All models use the same hyperparameters, including batch size of 3072.
7

---

Under review as a conference paper at ICLR 2023
Doc VQA
Widget
Text
Exact Match (%)
Pretraining
60
Captioning
Caps
40
Full
67.8
137.5
84.2
Variable Padded Stretched
Warmup 56.2
128.0
71.7
20
Masking
55.7
129.4
77.4
Screenshot Parsing
12.2
35.1
24.2
0
10k
20k
30k
Training steps (Warmup stage)
Ablations of pretraining components. Each ablation is a modification with respect to the full model, while keeping the total number of pretraining steps constant.
Figure 3:
Figure 4: Our puts prevent aspect-ratio distortion while minimizing padding.
in-
Table shows that all components of the pretraining scheme are crucial for good downstream task performance across all benchmarks. The biggest drop in performance comes from ablating the screenshot parsing stage, effectively reducing the pretraining to reading linear text. Ablating the warmup and masking is nearly equivalent on Voc and Widget Captioning while the warmup slightly more important in TextCaps. Overall, our results seem to indicate that reading and under- standing visually-situatee language a complex problem which needs a deep set skills including recognizing text, understanding language, and incorporating visual context.
Ablating rariubie-resolutionn inputs Figure compares various ways to convert input images into constant number of patches. This ablation is performed the warmup stage (Section 2.4) where we measure full sequence accuracy. The 'padded' variant maintains the original aspect ratio, but introduces significant padding, which sacrifices the effective resolution. The 'stretched' variant, typically used in ViT, introduces no padding but distorts the original image. Our variable-resolution inputs get the best of both worlds by maintaining the original aspect ratio while maximizing the budget specified by the sequence length 66 Experiments show that this benefit leads to more effective learning, even for task simple transcribing text the input image.
DISCUSSION
In this section, we lay out some of the challenges in training general-purpose visual language under- standing models, and discuss road map for future work.
Resolution Like Donut, we found that pretraining and finetuning performance are extremely sen- sitive the input resolutions X The difficulty in using high-resolution images has been bottleneck for pixel-only models since higher resolutions often lead to longer sequence lengths. This bottle- neck has in part been responsible for the dominance of OCR-based pipelines which are able to use lower image resolutions due to dedicated text e However, steady progress with Donut and Pix2Struct ruct combined with recent progress in long range transformers (Press et al. 202TI) provides hope that pixel-only models will bridge the gap with OCR-based pipelines.
The visual web As a first attempt towards a general-purpos visual language understanding model, focused on simplicity both in terms of how we use the HTML source and our choice for the pretraining corpus, C4-a known public corpus used in previous work (Raffel et al., (2020) that is significantly smaller and narrower than corpora used to train the largest language models today. However, web data includes even richer multimodal signals such as videos and interactions. We posit that future versions of general-purpose visual language understanding models will benefit from better data curation. This opportunity also comes with a caveat: just like text-based models, we must be careful of harmful content on the web, which multimodal models would also be sensitive to.
Generality While we have focused on general pixel-only models, we do acknowledge that using OCR -ic-piiplines or metadata can be appropriate or even necessary in certain domains. For NLP, the scaling of pretrained text based models has led to not only simpler model architectures and preprocessing, but also emergent abilities on newer tasks which were hitherto considered far too
'Illustratoons our variable-resolution method and the standard stretching method is in Appendix B
"See Appendix B for a concrete comparison. *OCR pipelines, while noisy, often result in manageable sequence lengths for large-scale encoders.
8

---

Under review as a conference paper at ICLR 2023
A veneral-purpon model may also enable broader applications for visual language, e.g. filling in missing accessibility annotations (Zaag et al. 2021). The broader objective of this work is to bring pretraining for visually language understanding step closer to text-base counterparts and pave the way for similar benefits from data and model scaling.
difficult (Wei al. 2022
7 RELATED WORK
To the best of our knowledge, no prior work has pretrained and evaluated visually-situated lan- guage understanding model on tasks spanning all four domains of documents, illustrations, user interfaces, and natural images. _ We build prior work primarily focused on single domain and briefly highlight the similarities as well as the points of departure with respect to such work here.
Document understanding State-of-the-an models in this domain based on pipeline of an ternal OCR system and a model that combines images and OCR annotations (Appalaraju et al.) [021] Powalski et al. 2021 Xu eu al. (2021), inter alia. Prominent representatives are LayoutLMV3 (Huang et al. 2022). which uses a simplified Transformer- based architecture and losses that encourage patch OCR alignment. TILT (Powalski al.) 202T) pretrains text decoder and an image + OCR- output encoder followed by intermediate finetuning on multiple QA tasks. Pix2Struct is more closely related Donut and Dessurt (Davis et al., 2022). also image-to-text models without OCR at inference time; the main difference stems from our more powerful pretraining task from ground truth structures and resolution flexibility enabling transfer to a variety of visual language domains.
UI understanding Models in this group have focused solely on the UI domain and have been pretrained on data from mobile and web apps. While some models use image-only inputs (Liu et al. 2018: Chen et al.) (020), higher accuracy approaches also make use of the structures of view hierarchies and element annotations, e.g. UlBert (Bai et al. 202I), ActionBert (He et al. [021], VUT (Li et al. 202 although the structured metadata is known be noisy (Li et al. (2020a). The screen parsing task (Wu et al., 202I). while similar in name, is an amalgamation of pipelines over domain-specific structures that are not intended produce transferable representations.
Natural image understanding A variety of model architectures (Singh et al. 2019 Sidorov et al, 2020 Wang al., (020) and objectives (Yang et al. 202T) have been proposed for understanding natural images containing short segments of text (e.g. street signs). The predominant source of pretraining data has been image-caption pairs often in conjunction with the output of OCR (Chen iet al. 2022bb Yang et al., 2021). GIT2 (Wang et al. 2022a).. the pixel-only SoTA, learns from 12.9 billion image-caption pairs and is about 4 times larger than Pix2Struct- it outperforms our model significantly on natural images (TexiCaps) but underperforms on illustrations (OCR-VQA). PaLI benefits from using pipeline with OCR, obtaining higher performance on TextCaps. These methods have not been evaluated on more text-dense input domains.
Illustrations Models for illustrations have not been fully pretrained on large scale data, perhaps be- cause such data is not readily available. Some components of such models, e.g. T5 and TaPas (Eisen- schlos et al. 2020) used the VL-T5 and VisionTaPaa models of Masry et al. (2022) or LATr's OCR output encoder (Biten et al.) 2022) have been pretrained on digital-born eCR-ed documents. Our approach outperforms current SotA models, without relying on other intermediate structures.
Related models learning from markup structure MarkupLM (Li al. 2021b) and Web- former (Wang et al. (022b) learn encoders of HTML from web pages. HTLM (Aghajanyan et al.) 2021) and CM3 (Aghajanyan et al. 2022) are generative models of simplified HTML to enable pow- erful zero-shot prompting with text and natural images. Im2Tex (Deng et al. 2017) is conceptually the most relevant in showing that pixel-only parser be learned from freely-available pairs of markup and renders, but they do not focus on transferring this signal to wider applications.
Datasets We have selected set of datasets representing challenges in visually-situated language understanding in a variety of domains, but our selection is not aimed to be exhaustive. The DUE benchmark (Borchmann et al., (20TT) focuses on more limited domain of visual document under- standing (e.g. excluding natural images and UIs), but integrates more comprehensive set of tasks within that domain, including document layout analysis and academic papers.
Some prior approaches have been evaluated two of these domains.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zeutlemoyer. Htl: Hyper-text pre-training and prompting of language models. arXiv preprint arXiv:2107.06955, 2021.
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multi- modal model of the internet. arXiv preprini arXiv:2201.0755 2022.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and Manmatha. Doc- Former: End-to-end Transformer for document understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 993-1003, October 2021.
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. Uibert: Learning generic multimodal representations for understanding. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pp. 1705-1712, International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: /4496////aii.2022/233 URL httph:tt//o/oiii http///..........0.......
ijcai. 2021/235
Main Track.
Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and Manmatha. Latr: Layout- aware transformer for scene-text vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16548 -6558, 2022.
Lukasz Borchmann, Michal Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michal Turski, Karolina Szyndler, and Filip Graliński. Due: End-to-end document understanding benchmark. In Thirty-fifit Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Chongyan Chen, Samreen Anjum, and Danna Gurari. Grounding answers for visual questions asked by visually impaired people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19098-19107, 2022a.
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. Unblind your apps: Predicting natural- language labels for mobile gui components by deep learning. 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pp. 322-334, 2020.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali: A jointly-scaled multilingual language-image model, 2022b. URI. https:
//arxiw.org//as//2000 06794
Brian Davis, Bryan Morse. Bryan Price, Chris Tensmeyer, Curtis Wigington, and Vlad Morariu. End-to-end document recognition and understanding with Dessurt. In Text in everything ECCV workshop, 2022. httpstts::// aattps://arxiv. wwwpxxiw..oog....oooooero 2203. 16618
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M Rush. Image-to-markup generation with coarse-to-fine attention. In International Conference on Machine Learning, pp. 980-989. PMLR, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v//N19-1 1423. URL https:
7/acl aochlantholoooo ogy. htwphhhhoooyyo..g/111l1/.11 N-
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom,, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Julian Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with interme- diate pre-training. In Findings of the Association for Compulational Linguisties: EMNLP 2020, pp. 281 281-296, Online, November 2020. Association for Computational Linguistics. doi: http://www22000..dnnnnnnn-me 1ircinggs-emnp..2
URL /ttpe://aclantboooo..orzz org.0o0..
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017,
Zecheng He, Srinivas Sunkara, Xiaoxue Zang, Ying Xu, Lijuan Liu, Nevan Wichers, Gabriel Schu- biner, Ruby Lee, and Jindong Chen. Actionbert: Leveraging user actions for semantic under- standing of user interfaces. In 35th AAAI Conference on Artificial Intelligence, AAAI 2021, 35th AAAI Conference on Artificial Intelligence, AAAI 2021, pp- 5931-5938. Association for the Advancement of Artificial Intelligence, 2021. Publisher Copyright: Copyright @ 2021, Asso- ciation for the Advancement of Artificial Intelligence (www.aaai.omm org). All rights reserved.; 35th Conference date: 02-02-2021 Through
AAAI Conference on Artificial Intelligence, AAAI 2021; 09-02-2021.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutLMv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In European conference on computer vision, pp- 235-251. Springer, 2016.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without OCR. In ECCV, 2022. URL httpt://aaxxxxi httg//bs//1/........////
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequeece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, July 2020. doi: 10.1865.1 h/tp:////2200ll11111 703. URL httpp:://aaaammmhoooyy...oo org.20/00o.y...22.... acl mai inn. n.n.7. 703
Chenliang Li, Bin Bi, Ming Yan, Wei Wang. Songfang Huang, Fei Huang, and Luo Si. Structurallm: Structural pre-training for form understanding. arXiv preprint arXiv:2/05.1 1/2100 2021a.
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language for visually-rich document understanding. arXiv preprint 1 2021b.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8198-8210, Online, July 2020a. Associ- ation for Computational Linguistics. doi: 10.18653/v 1/20330.112222ll-11al1 mcinn...799. 7/www. aclweb. wwg.oorg///aoooo ant
URL https:
acl ma main....................mm...m 729
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning:
Generating natural language description for mobile user interface elements. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5495- 5510, Online, November 2020b. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.emnlp-main.444 443. URL htths:/:::// //acl antho org. wrgp//g/./200/ eww.20000001110000.mm.m. p emn i 443
Yang Li, Gang Li, Xin Zhou, Mostafa Dehghami, and Alexey Gritsenko. Vut Versatile transformer for multi-modal multi-task user interface modeling. arXiv preprint 22 2021c
11

---

Under review as a conference paper at ICLR 2023
Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semantics for mobile apps. Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology.22 2018.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: https://aclaane
100p:/2////2222...nn.no..... ogy. 2022. Findings-aa
URL
177
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision,
pp. 200022002222 200002209,2200 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947- 952.. IEEE, 2019.
Rafat Powalski, Lukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michat Pietruszka, and Gabriela Palka. Going full-tilt boogie on document understanding with text-image-layour trans- former. In International Conference on Document Analysis and Recognition, pp. 732-747. Springer, 2021.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12 12409, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. OpenAl, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jml wttpppappss/////200-00o 074 html
Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky. Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. arXiv preprint arXiv:2207.0699) 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556 2565, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v 10/65//1/P118/ 1238. URL Bollhttts::/aaaatttoooo htt.pp////aalathoooooooooo///// 1238
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604. PMLR, 2018
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. In European Conference Computer Vision, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolu- tion discrepancy. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bur E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL htteltttss//// / wtwp::/wwww..ee....oooi.....o..o http:///wwaaaaaaaoaoaooooeeoepap
12

---

Under review as a conference paper at ICLR 2023
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Au- tomatic mobile ui summarization with multimodal learning. In The 34th Anmual ACM Symposium on User Interface Software and Technology. UIST 21,, pp. 498-510, New York, NY. USA, 2021a. Association for Computing Machinery. ISBN 9781450386357. doi: 10.1145/3472749.3444777777
URL htt.:https///ooooo. http::::/dooo.rgg////// 0005//001111544712166 0341/44477799.37766..
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022a.
Jing Wang. Jinhui Tang, and Jiebo Luo. Multimodal attention with image text spatial relationship for ocr-based image captioning. In Proceedings the 28th ACM International Conference Multimedia, pp. 4337-4345, 2020.
Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information extraction. In Proceedings of the ACM Web Conference 2022, pp. 3124-3133 2022b.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. CoRR, abs/2108. 10904, 2021b. URL
pictss:/aarae
hw.gg/abs.. abs/ awws//18..... 211..1111..1111111
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint wrXiv:2206.07682 2022.
Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey Bigham. Screen parsing: Towards reverse en- gineering of UI models from screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology, pp. 470-433, 2021.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) 2021, 2021.
Zhengyuan Yang. Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8751-8761, 2021.
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, et al. Screen recognition: Creating accessibility meta- data for mobile applications from pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2021.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.
13

---

Under review as a conference paper at ICLR 2023
A
DATASET DETAILS
Dataset
Domain
Description
OCR-VOR VQA
Illustrations
VQA over book covers.
ChartQA
Illustrations
VQA over charts (visualization of tabular data)
A12D
Illustrations
VQA over science diagrams
RefExp
Uls
Detect UT component matching natural language query
Widget Captioning
Uls
Captioning UT component screen
Screen2Words
Uls
Captioning screen to describe functionality
Tex:Caps
Natural images
Captioning of natural images containing text
VQA
Documents
VQA over scanned documents.
InfographicsvQA
Documents
VQA over high-res infographics.
Table 2: Summary our proposed diverse benchmark for visually-situated language understanding
B RESOLUTION IN VISUALLY-SITUATEE LANGUAGE UNDERSTANDING TASKS
Aspect-ratio
Aspect-ratio
preserving
distorting scaliing
scaliig
patches
patches
Variable resolution imputa (oura)
Fixed resolution imputs (standard VIT)
Figure 5: Comparison of our variable resolution inputs and the typical fixed resolution input. We illustrate the preprocessing for a target sequence length of 36 patches for both inputs.
Effective number of pixels
Effective number of pixels
2
2
2
218
2
20
221
2
2
2
2
2
2
20
Pix2Struct-hase
300
Pix2Struct-large
Examples per second
60
250
ANLS (%)
200
150
P Pix2st ruct-base base
Donut
100
20
VIT pretraining
50
VIT finetuning
128
256
512
1024
2048
4096
8192 16384
128
256
512
1024
2048 4096
Input sequence length
Input sequence length
Figure 6: Overview of the impact of resolution on the DocVQA task. Note that the bottom axis only applies to Pix2Struct. Pix2Struct is also the only model that adapts to various res- olutions seamlessly, without any retraining or post-hoc parameter creation. (Left) In both Donut and P Pix2struct, measured by auto-regressive decoding (max decoding length of 32 tokens) on the validation set of DocVQA using a v3-8 Cloud TPU.
show clear benefits from larger resolutions. (Right) Inference speed
Previous methods rescale input images to fixed resolutions, which can introduce aspect ratio distor- tions that be severe for inputs such webpages and documents. In contrast, prevent aspect ratio distortion by rescaling imput images up or down such that we extract the maximal number of patches that fit within the given sequence length (Figure
14

---

Under review as a conference paper at ICLR 2023
Figure 0 gives overview of the importance of input resolutions in visually-situtted language understanding tasks. Though 1ex2Struct is more efficient at making use the imput resolution, both Fix2Struct and Donut require high resolutions to perform well on Doc (note the log scale). For example, we only see significantly diminishing returns after about 1M pixels (4096 patches of 16 x 16 pixels for P rusttruct and 1024 x 1024 for fixed-resolution models). However, ViT models typically pretrain with resolutions of 224 x 224 and finetune with up to 512 x 512. This subtle but critical detail that makes using standard ViT out of the box suboptimal.
On the right of Figure 6 we also present example inference speeds on a v3-8 Cloud TPU when performing inference on Doc At full resolution (4096 sequence length or 1M pixels), the base model processes 62 documents per second, and the large model processes 20 documents per second.
C
FULL RESULTS
Chart QA
OCR
Ref Exp
Widget Cap
Screen2 Words
Text Caps
Doc
Info VQA
Method
A12D
VQA
VQA
TILT
87.1*
VUT
94.8
64.3
TAP
99.5
LATr
67.5
PLC
97.0
Pipelined
T5 + 2D + U
81.0
46.1
RoBERTa
69.5
LayoutLMv3
83.4
DQA-NET
38.5
UI Bert
90.8
M4C
63.9
81
14.7
VasionTaPas
45.5
PaL.I
160.4
GIT2
70.3*
145.0
Pixel only
Donut
41.8
30.8
66.0
127.4
56.4
74.4
67.5
11.6
Pix2struct-Bas
56.0
40.9
69.4
92.2
133.1
107.0
88.0
72.1
38.2
Pix2scruct-Large
58.6
42.1
71.3
94.2
136.7
109.4
95.5
76.6
40.0
Table 3: Amongst single-task single-model methods, P x25t achieves state-of-the-at results
on out benchmarks spanning 4 domains. indicates that the method used additional labeled data from other tasks and are not directly comparable to single task methods. VasionTaPas uses table extraction tool. DQA-NET uses diagram processing tools for detecting arrows, blobs, etc in addition standard OCR. UI Bert and VUT use Android view hierarchies. All other non-image methods use standard OCR.
Table reports full results for pipeline and pixel-only methods across all datasets. For fair com- parison and case of experimentation, we focus on single-model and single-task baselines trained on standard splits. Several (per-task) SotA (Li al., 2021c| Masry et al., 2022) inputs (e.g. view hierarchies for UIs or gold data tables for charts) making it difficult to apply them
other domains.
D HYPERPARAMETERS
Finetuning The base and large models are finetuned with input sequence length of 4096 and 3072 respectively, except the base model on InfographicVQA which benefits from a longer sequence length of 6144. We cannot ory constraints. We finetune for 5000 or 10000 steps with a batch size of 32, 128, or 256, with hyperparameter tuning and early stopping based on the validation set. Table hyperparam- eter values for all tasks.
longer sequence length for the large variant due TPU/GPU mem-
15

---

Under review as a conference paper at ICLR 2023
Base
Large
Dataset
Seq Len
Batch
Steps
Seq Len
Batch
Steps
Doc
4096
256
10000
3072
128
10000
Infographic VQA
6144
64
10000
3072
128
10000
AI2D
4096
32
5000
3072
32
5000
ChartQA
4096
256
10000
3072
128
10000
OCR-VQA
4096
256
10000
3072
128
10000
RefExp
4096
256
10000
3072
128
10000
Screen2Words
4096
32
10000
3072
32
10000
Widget Cap.
4096
256
5000
3072
128
5000
TextCaps
4096
256
5000
3072
128
5000
Table 4: Model hyperparameters
E WARMUP STAGE EXAMPLE
The elves, it seemed, were possessed of some mysterious power over the arts; without eve
The eives, it seened, were possessed of some mysterious power the arts; without
Figure Example of input-output pairs during the warmup stage.
Exposing the model to short, intense "warmup" stage of simply learning to read, results in a strong curriculum learning effect where (1) pretraining is more stable and converges faster, and (2) we observe better finetuning performance. 7 an example of rendered text from the Books corpus with its "parse".
F PRETRAINING EXAMPLES
The figures below show screenshots of our pretraining data along with ground-trut -ruth and predicted parses.
16

---

Under review as a conference paper at ICLR 2023
THLINDERHAWK
Bring your child in-for drop-in to get
Miiting from out of town transling dLb sports, child's routine is not disrupted, Bring them
show you around
Ground-truth Parse
<<<<CrossFit Thunderhawk Rio Rancho>
<dedicated promote healthy kids and teens in Rio Rancho, NM>>
<<Home> <About> <Schedules <Media> <Blog> <Contact Uss <Free Class>>> <<rrop-ins>
<Bring your child in for drop-in to get a WOD in!>> <<IIf you are visiting from out of town traveling for club sports, make your child's routine is not disrupted. Bring them in for a drop in to get a WOD in!?
<<l-day CrossFit Athlete $15>
<l-day Compotitor $25>>>
<<Become Member>
<We'd love meet you and show you around.>>>>
<<<<ing <<<<ing_src= www.thhnnerraaat-ogo-------- Predicted Parse
Lng_allt-Tunderraaw Sports & Fitness>
<Thnderhawk Sports & Fitness>>
<<Home> <About> <Frograms> <Team> <Blog> <Contact Us> <Get Started>>> <<<Drop-Ins>
<Bring your child in for a drop-in to get a workout>>
<<<If you are visiting from out of town or traveling for club sports, make aure your child's routine is not disrupted. Bring them to our drop-in for full session!> <<l:1 drop-in for
17

---

Under review as a conference paper at ICLR 2023
The Chocolate Cult
Helping Chocolate 2009
Pnterary 2013 Cocoa Sweet Dough Hearts
Todby sharing secipe, Recipe using BaleNSary Saveel Dough ofher chocolcte chipo and chocolato spread. Netthrr worked great. Today, in conjunctioon with the 2019 Good Cookie Food Bloggar Valartinels Event, bridd somemn Jaminns's Jamin's theme help too As
som-hoomemado
loday going free produst that was cent to my by Rhoces DakeNSSer exchange for les you no other fon
[X]
Dough By lammyJo edkhart
Ingredients:
loaf frozen sweet dough T whole wheat flour
dark
Diradions:
dough according directons.
2. Dust tho surfoce whore you will tut dough with the flour and the cocco, uso your hand mix thom togother.
Ground-truth Parse
<, tried something Valentine' a thened. If you' d like to help raise money for fighting children's cancer you above and help out, too. inspiration for this semi homemade recipe, looked at the two recipes on the bag of sweet dough, I got an idea and today I'm going to share with you how that worked out. \xa0 I got the bag of Sweet Dough using coupon for free product that their products and sharing the results with all you; no other form of compensation was received.>
follow the link right
my by Rhodes BakeNServ in exchange for testing out
Predicted Parse
<, tried something Valentine's thaned. If you' d like help out, I think you'd d go right ahead and do post. Click on the link right above and help out, too. I've shared up two recipes on the bag of sweet dough. I got an idea and today I'm going to share with you the second one.
inspiration for this chme-maeeaade recipe,
Thank you for any the amazing baking ideas plus this free product that was sent to my by Rhodes BakeNServ in exchange for testing. I'm really excited and sharing this recipe with all of you
18

---

Under review as a conference paper at ICLR 2023
Fringe Benefits
Polarized vs. UV
Fellss underiede rebly theees sesson for feshion for a mulliiiuee of reestrss
Ground-truth Parse
<<<1088 FEMALE 100% UV PROTECTION SINCE 1999> <FAST FREE SHIPPING>>
<ing_alt-Velvet Ryewear>
<<<<Fringe Renefits>
<<Posted by> <Lindsay Sperin> <on> <Agusst 19, 2016>>>
<<img_ soo=ingrrmmg>>
<Fall is undeniably the best season for fashion for multitude of reasons.>
<img_arc=img>>> <<NENS>
<<Polarized vs. UV Protection What's s The Difference?> <What's Hot in The Hamptons>>>>
<<ing_src=en-us img all==e>>>> <English>>>
Predicted Parse
<<<00 OFF YOUR FIRST ORDER WITH CODE: FIRST10> <FAST FREE SHIPPING>> <ing alng_aat=Veleet>> <<<<Fringe Benefits>
<<Posted by> <Velvet Fashion> <on> <October 1, 2018>>> <<Fall is undeniably the best season for fashion for a multitude of reasons.>
alt=Frinee Benefits>>> <<search>
<cPolarized vs. UV Protection: velvet' Best Sunscreen> whe Best Sunblock Sunacreen>>>>>
19

---

Under review as a conference paper at ICLR 2023
RISE AGAIN.
Masare GC Smic growing universe. Eccouse
grow spin games that Uillzo btog8ros games sridinng
Ground-truth Parse
<<Menu>
<ing ting__rr=ftg_webhaade <<<spin-Off Games>
<<Fairytale Games is a growing universe. Because of this, we have and will continue to grow spin-off games that utilize characters, games. Keep checking back and
storyl ines, and even poke fun you just might be surprised at what you see!>
<<Rumleettiltskin.
<Super Fairytale Fighters 2>>
<<<Share this:>
<<Twitter> <Facebook>>>
<Loading...>>>>
<<Leave Reply>
<<<Your email address will not be published.> <<Required fields marked> <*>>>
<<Comment> <*>>>>>>
Predicted Parse
<<Menu> hhng_aerccrropee--ioohhhaae
<<<Faizytale Games>
<<Fairytale Games is growing universe. Because this, excited continue grow spin-off games that utilize characters, storylines, and even poke fun of our games. Keep checking back and you just might be surprised at what you see!>
<<Faizytale Games>
<Faizytale Games Steam>>
<<<Share this:>
<<Twitter> <Facebook>>>
<Loading...
<<Leave Reply>
<<<Your email address will not be published.>
<cRequired fields are marked
20

---

Under review as a conference paper at ICLR 2023
Caronavirus Upéate! We are open and roady help you. conducting our appointmmntts pitone help prevent
STAMPS & STAMPS
& Contact for free Initial Consultation
Abcl
Testmon els
Tax Rele!
Contact
FAMILY LAW
We
stressful
disput
Ground-truth Parse
<<<<Coronavirus Update! We are open and ready to help you.> <We are conducting most of our appointments via phone to help prevent the spread of the virus.>>
<Chapter 13 Coronavirus Update>> <<ing_ere=Logoo img_alt=Stamps & Stamps Attorneys Law>
<ing_aro=Phone>
<Contact for free Initial Consultation>
<<Cal1 Ua> <997) 247-6447>> <<Text Us> <997) 265-6418>>>
<<Hone> <About> <Articles> <Videos> <Testimonials> <Tax Relief> <News> <Podcasts> <Rate Us> <Contact>>
<<We can provide the guidance you need to get through stressful family> <disputes with your rights and interests intact.>>
Bangalty Overview>
<<Bankruptcy> <overview>>>
wongssoc-rrmminl--eeeeee ing alt-rrinnlll Defense & Traffic Offenses>>>
Predicted Parse
<<<<Coronavirus Update! We are open ready to help you.> <We are conducting most of our appointments via phone to help prevent the spread of infection.>> <CLICK HERE FOR MORE INFO>> <<ing_arc==ogo ime_alt=St.aps stamps Attorneys Law>
_rng_arc=phone>>
<<<call Us> <9904) 222-2222>>
<<Text Us> <(904) 222-2222>3>>
<<Home> <About> <Artici es> <
21


Reviews:
[{"review_id": "x9b9XESt2G", "reviewer": "Reviewer_GpRK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a pretrained image-to-text model for visual language understanding, which has a wide range of application in sources such as textbooks with diagrams, webpages with images and tables, and mobile apps with buttons and forms. The model has a simple and general architecture which only takes an image as the input. It is pretrained by parsing masked screenshots of webpages into simplified HTML, which subsumes common pretraining signals. This paper also proposes an integration of language and vision inputs for fine-tuning the model. The model achieves SOTA results in six out of nine tasks across four domains.\n", "summary_of_the_review": "This paper proposes a simple pretraining framework for visual language understanding. The simplicity of the model, taking only image as the input, enables it to apply to diverse tasks and achieve supreme results. It is an exciting innovation in this area, and I believe it would greatly benefit the community if the data/model can be made public.\n", "strength_and_weaknesses": "Strength:\n\n1. Pix2Struct uses a general-purpose pixel-to-text design, which simplifies the model architecture and can be easily applied to multiple domains.\n2. The masked screenshot parsing objective is intuitive and effective. The pretraining data is easy to obtain from the web. The warmup stage with an image-to-text curriculum further improves the pretrained model.\n3. The proposed fine-tuning strategy seamlessly integrate language and vision inputs by rendering language prompts on the image, without changing the architecture.\n4. A single pretrained model achieves strong performance on multiple visual language understanding tasks from diverse domains.\n\nWeakness:\n\n1. The presentation of this paper is easy to follow, but not clear enough. Many important techniques are described at a high level, for example \n\n    (1) How the proposed variable-resolution input representation is implemented, how to determine the number of patches for width and height, do small and large inputs have the same number of patches?\n\n    (2) How the pretraining data is collected and processed, especially for the warmup stage. \n\n    (3) How the masked parts are selected?\n\n2. The authors do not mention if the data/code/model will be publicly available. It would be difficult for the community to reproduce or follow if they cannot be released.\n\n3. Some of the designs lack empirical justification. For example, are there alternative ways to fine-tune the model, such as concatenating image and text? Does the location/size/font of where the prompt is rendered affect the performance? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "nTX6MEbqgNW", "reviewer": "Reviewer_Apqp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes screenshot parsing as a simple yet effective pre-training task for visually-situated language modeling (e.g., document QA). The model gets rid of the OCR and directly accepts an image (with rich layouts and rendered text) as input. The empirical results show that the pre-training outperforms prior pixel-only methods by a large margin.", "summary_of_the_review": "Overall, the paper shows that screen parsing is an effective pre-training task for visually-situated language modeling. I am on the fence about the design choice of using a pure pixel-only transformer but that does not undermine the core argument for screen parsing. ", "strength_and_weaknesses": "The paper makes a welcomed and impressive empirical contribution, with several nice discussions on certain design choices. I will note 3 concerning points:\n\n1. OCR or no OCR?\n\nThe paper is in line with the trend of replacing domain-specific components with a general pre-trained neural model, which is generally welcomed. However, in domains where OCR is cheap and adequate, I am not sure if it is worthwhile to spend so much compute on replacing the OCR with a pixel-only LM. \n\nAn ideal solution would be an adaptive model which is pre-trained on such general screenshot parsing task but when adapted to a downstream task, could be fine-tuned to accept domain-specific inputs when there are cheap and easy-to-use solutions. \n\n2. Scaling up the model parameters seems to only bring marginal performance improvement.\n\nIs there any observation about this? Is it because the pre-training task is too simple? It would also be good if the authors could provide evaluations on the pre-training task itself, using some simple metrics such as blue or exact match. \n\n3. More details on how the pre-training data are extracted and cleaned should be provided. \n\nThe authors mentioned that the data are from C4. Does C4 already provide the cleaned data in HTML? My impression of C4 was that it was primarily a web-text corpus.\n\nStatistics on the pre-training data should also be provided. E.g., what’s the lengths distribution of the output strings?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FEImxIqens", "reviewer": "Reviewer_dECs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Pix2Struct, a pretrained image-to-text model. The model is pretrained to parse masked screenshots of web pages and the authors show that it can be fine-tuned on multiple tasks containing visually-situated language (VQA, Image captioning, Infographic VQA). The fine-tuned models achieve state-of-the-art results in six tasks, while using only pixel-level inputs. The authors also introduce a variable-resolution input representation to prevent the distortion of input images.\n", "summary_of_the_review": "Although the proposed method for pre-training a model from web page screenshots is interesting, the conditions of the experiment, with very large amounts of data and mobilizing unreasonable computational resources, do not allow to highlight its interest, its applicability and the reproducibility of the results. ", "strength_and_weaknesses": "Although interesting and very well written, this paper does not make major contributions to the field. \n\nThe first contribution concerning the pre-training strategy is quite interesting. The authors propose a strategy based on web pages screenshots that allows to collect quickly and easily a very large amount of data. Moreover, these data can be quite varied and contain a large variety of elements, which allows to train a model that can work on many tasks. \n\nThe second contribution consists in adding modifications to the transformer inputs to handle variable aspect ratios and resolutions. This strategy, although interesting, brings little gain to the results. Moreover, it does not constitute a major contribution to the paper.\n\nFor the experiments, the following comments should be addressed.\n\n1. The idea of simplifying the multi-modalities problem is interesting. However, it could have been nice to compare the method to a standard approach with a modality combination.\n2. The input masking is done using crossed-out opaque bounding boxes. There is no justification about this choice. I wonder if this choice has an impact on the results compared to other approaches like removing the boxes or using white bounding boxes.\n3. The authors claim that the changes they added to the standard ViT inputs provide major advantages in terms of robustness to extreme aspect ratios and on-the-fly changes to the sequence length. However, except for the results presented on Figure 4, there is no experiments showing these advantages.\n4. It is not clear whether the modifications applied to the screenshots and HTML only consist in masking some texts or if other modifications are applied.\n5. I would have appreciated to have the inference times of the base and large models.\n6. The method of Wang et al (2021a) for the Screen2Words task could be more detailed.\n7. It is not said on which task and dataset the ablation study on the inputs resolution is carried on.\n8. 13 articles are cited as ArXiv preprints, please update the citations for articles that have been published.\ninor comments:\n\n1. In Table 2, it would make the table clearer to add a line with the metrics.\n2. In Table 2, the value 160.4 should be highlighted instead of 145.0.\n3. The values should be given with the same number of significative numbers:\n    - 11.27 in AI2D\n    - 145 in TextCaps\n    - 40 and 81 in InfographiVQA\n4. Figure 3 is a table, it should be renamed Table 3.\n5. Typo:\n    - Figure 4 caption: \"Our variable-resolution inputs prevent**s**...\"\n6. In the appendix\n    - In the fine-tuning section, it is said \"Tables 4 and 4\", it should be corrected to \"Tables 4 and 5\". By the way, tables 4 and 5 are quite similar, they could be merged into a single table.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject