Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models
Authors: Geewook Kim
Hodong Lee
Daehee Kim
Haeji Jung
Sanghee Park
Yoonsik Kim
Sangdoo Yun
Taeho Kil
Bado Lee
Seunghyun Park
Venue: EMNLP 2023 Main (2023)
Keywords: Visual Document Understanding
OCR
Contrastive Learning
Multimodal Language Models
Document Visual Question Answering
Transformer-based Models

Abstract: Recent advances in Large Language Models (LLMs) have stimulated a surge of research aimed at extending their applications to the visual domain. While these models exhibit promise in generating abstract image captions and facilitating natural conversations, their performance on text-rich images still requires improvement. In this paper, we introduce Contrastive Reading Model (Cream), a novel neural architecture designed to enhance the language-image understanding capability of LLMs by capturing intricate details that are often overlooked in existing methods. Cream combines vision and auxiliary encoders, fortified by a contrastive feature alignment technique, to achieve a more effective comprehension of language information in visually situated contexts within the images. Our approach bridges the gap between vision and language understanding, paving the way for the development of more sophisticated Document Intelligence Assistants. Through rigorous evaluations across diverse visually-situated language understanding tasks that demand reasoning capabilities, we demonstrate the compelling performance of Cream, positioning it as a prominent model in the field of visual document understanding. We provide our codebase and newly-generated datasets at https://github.com/naver-ai/cream.

Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models
Geewook Kim 1.2+ Hodong Lee Dachee Kim
Haeji Jung
Sanghee Park
Yoonsik Kim Sangdoo Yun't
Taeho Kill
Bado Lee
Seunghyun Park
NAVER Cloud AI 2KAIST AI
Korra University
NAVER AI Lab
Abstract
EXPLORE
Recent advances in Large Language Models (LLMs) have stimulated surge of research aimed at extending their applications to the visual domain. While these models exhibit promise in generating abstract image captions and facilitating natural conversations, their per- formance on text-rich images still requires provement. In this paper, we introduce Con- trastive Reading Model (Cream), novel neural architecture designed to enhance the language- image understanding capability of LLMs by capturing intricate details that are often over- looked in existing methods. Cream combines vision and auxiliary encoders, fortified by con- trastive feature alignment technique, achieve more effective comprehension of language information in visually situated contexts within the images. Our approach bridges the gap between vision and language understanding, paving the way for the development more so- phisticated Document Intelligence Assistants. Through rigorous evaluations across diverse visually-situe situated language understanding tasks that demand reasoning capabilities, demon- strate the compelling performance of Cream, positioning as a prominent model the field of visual document understanding, We provide our codebase and newly-generated datascts at https://gil / thub. com/naver ai/cream.
U.J.
Q: What the name of the store that likely sells this
The name of the store that likely sells boar in this is "Joo''".
LLaVA
explore
X
Besed on the provided text, there no spacific mention of store that likely selts bear the ad.
+ + LLM
HOUSE
Figure I: Comparison on a text-rich image. The pro- posed method, Cream, precisely interprets and reads the relevant store's name from a poster containing multiple text instances and visuals, overcoming limitations of existing approaches (e.g., OCR+ChaiGPP)) Our Cream efficiently extracts visual features from the image, thus enabling LLMs to provide an appropriate response.
1 Introduction
LLMs, particularly Large Visual Language Mod- els (LVLMss, leveraging vision encoders designed to tackle information-ride visual tasks (Radford et al., 2021a; Tsimpoukelli et al., 2021; Wang et al., 2022a; Alayrac et al., 2022; Wang et al., 2022b; Driess et al., 2023; Zhu et al., 2023).
Recent advances in Large Language Models (LLMs) (Brown et al., 2020; OpenAl, 2023: Zhang al., 2022; Touvron et al., 2023) have facilitated the development of numerous real-world applica- tions. Researchers are increasingly focusing on extending these unimodal LLMs to multimodal
To evaluate LVLMs, various downstream tasks have been employed, such as image captioning. visual dialogue, grounding, reasoning, and ques- tion generation. Although LVLMs demonstrate im- pressive results in these tasks, a recent study (Liu
Core contributions. Conresonndecee Geewook Kim: gukim. rsrchegmail. com.
Work done during internship at NAVER Cloud AL. Advisory contributions. A description of each author's contribution is available at the end of this paper.

---

al., 2023b) argued that LVLMs exhibit limita- tions when dealing with visual tasks on text-rich im- ages, leading to reduced applicability in real-world applications, such as Document Visual Question Answering (Document VQA). Document VQA tasks require the comprchensive analysis of mul- tiple types of information, including text, objects (e.g., figures charts), and layout. However, ex- isting LVLMSs struggle deliver satisfactory so- lutions due their limited ability to extract fine- grained features from images, as shown in Figure I,
VDU tasks.
We suggest a contrastive learning-based train- ing technique to improve performance and enhance robustness.
We offer accessible approach that allows for integrating the proposed model into LLMs, thus expanding the versatility of LLMs in mul- timodal domains by providing rich visual in- formation.
In this paper, we introduce Cream, Contrastive reading model, specifically designed to effectively overcome these limitations. Cream features streamlined and practical architecture that seam- lessly integrates general-vision encoder with aux- iliary encoders and novel training techniques. In addition to a primary vision encoder for overall visual feature extraction from document images, Cream employs auxiliary encoders such as OCR and object detectors- text and object-specific feature extraction. Cream handles fine-grained fea- tures without missing image details while under- standing the visual context. When combined with LLMs, Cream overcomes the limitations of LVLMs and achieves robust performance on text-rich im- ages. To further enhance the model, we propose a contrastive feature alignment method to balance disparities between the vision and auxiliary fea- tures extracted from each encoder during training.
We demonstrate the proposed method's su- perior performance on challenging text-rich VDU benchmarks via rigorous experiments.
We contribute valuable resources to facilitate ongoing research and development in VDU tasks by sharing our codebase and newly- generated datasets.
2 Related Work
2.1 Applying LLMs to Visually-Situated NLU
Large language models (LLMs) have demonstrated outstanding performance across various applica-
tions (Rae et al., 2021; Brown et al., 2020; Chowd- hery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023), demonstrating their ability to generate responses in accordance with user intent (Ouyang et al., 2022; Shahriar and Hayawi, 2023). Re- searchers have explored the use of LLMs in tasks related visual understanding (Tsimpoukelli al., 2021; Alayrac et al., 2022; Zhu et al., 2023; Liu et al., 2023a; Ye et al., 2023); however, their ap- plication visually- derstanding (NLU) tasks, such as question answer- ing on text-rich document images, remains limited. Previous approaches have attempted to integrate visual embeddings OCR tokens into LLMs address this gap (Li al., 2023; Dai al., 2023; Wang et al., 2023). However, these methods suffer from inefficiency, as they require numerous LLM tokens and entail considerable computational over- head. For instance, documents in DocVQA (Tito et al., 2021) consume an average of 400 and maxi- mum of nearly 3K OCR tokens. To overcome these obstacles, propose the integration of Cream, which imparts language-image understanding ca- pabilities to LLMs using soft visual prompts with fixed size. This integration has the potential enhance the efficiency and accuracy of LLMs visually- NLU tasks.
We conduct extensive experiments on challeng- ing text-rich Document VQA benchmarks. We perform experiments on models: our stan- dalone Cream model and model that combines our Cream model with frozen LLMs. The experi- mental results demonstrate that standalone Cream achieves results comparable to the state-of-the-art in tasks that necessitate the extraction of specific text information from document images. Further- more, observe that when combined with LLMs, Cream demonstrates robust performance in Visual Document Understanding (VDU) tasks, which are challenging for existing LVL.Ms. Lastly, we will open-source Cream's 's codebase and the newly built VQA datasets to foster further research and innova- tion in the field of visual document understanding. Our contributions are summarized as follows:
natural language un-
We present novel neural architecture and as- sociated model training techniques tailored for text-rich image understanding tasks, highlight- ing improved performance across challenging

---

2.2 Visual Document Understanding
3.1.1 Architecture
Visually-situated NLU blends computer vision and NLU techniques to accurately analyze visual data through language. Early approaches emphasized OCR and leveraged extracted text for contextual analysis (Xu al., 2020; Hong et al., 2022). Some contemporary methods directly processed docu- ment images, circumventing the need for exter- nal OCR models (Kim et al., 2022; Davis et al., 2023; Lee et al., 2022; Liu et al., 2022), while others leveraged both image and OCR- OCR-extracted text for improved performances (Kil et al., 2022; Tang et al., 2022: Appalaraju et al., 2021; Xu et al., 2021; Huang et al., 2022). LayoutLMMv3 (Huang al., 2022) introduces the word-patch align- ment technique, which classifies the relationship between text and corresponding image patches. UDOP (Tang et al., 2022) employs a unified en- coder that handles both image and text features, transforming the information from both modali- ties into vision-text embeddings by summing the image patch and text features. This strategy sures alignment between modalities, such image patches, tokens, and layout information, by fus- ing multimodal features within single encoder. contrast, Cream extracts fine-grained, aligned multimodal features through contrastive learning (CL), climinating the need for the fusion encoder. The CL approach prevents ever-fusion of informa- tion from each modality, allowing the decoder to effectively utilize the multimodal semantics inher- ent in visually-rich documents. Moreover, due to this architectural design, Cream exhibits enhanced robustness to OCR errors.
Cream is composed of two encoders and one de- coder. The vision encoder computes vector rep- resentations of the input image patches. Concur- rently, feature evidence, such as text or general object information in target images, extracted in text format and encoded by an auxiliary text en- coder. The embeddings from both encoders are then aligned using our proposed CL scheme (see Figure 2). The aligned features undergo further processing by the decoder to extract necessary in- formation. Additional details will be provided in the sections that follow.
Vision Encoder The vision encoder converts the input image HxWC into embeddings {z, , size or the number of image patches, and d is the dimension of the output vectors. We can use CNN- based models (He et al., 2016) or Transformer- based models (Dosovitskiy et al., 2021; Liu et al., 2021) as the encoder network. In this study, we em- ploy Vision Transformer (Dosovitskiy et al., 2021) with 2D absolute position encoding (Xu et al., 2020) and variable-rcsolutionn mechanism (Lee et al., 2022) to simplify the process. The variable- resolution mechanism ensures a constant number of patches without distorting the original image aspect ratio.
where is the feature map
Auxiliary Encoder The auxiliary encoder en- codes the extracted feature evidence, such as OCR boxes and general object boxes, into embeddings (2:)2, lęigń), where ñ is the maximum se- quence length of the encoder. In Figure 3, the feature evidence converted into token embed- dings using the recognized text for the OCR boxes and the recognized semantic object label for the general object boxes. A type embedding is added to differentiate between OCR and general object boxes, and 2D absolute position encoding is ap- plied for encoding the location information. We employ the encoder of BART (Lewis al., 2020) as the Auxiliary Encoder.
3 Method
Our goal is to develop a system that accurately an- swers natural language questions based on specific evidence in an input image. To achieve this, we propose model that explicitly identifies feature evidence in the image, such as texts and objects.
3.1 Contrastive Reading Model
We introduce Contrastive reading model (Cream) with two potential application scenarios: (i) func- tioning independently, where decoder module of Cream directly generates desired text infor- mation, and (ii) operating in conjunction with an LLM, serving to provide soft visual prompts for the LLM. A comprehensive overview of the entire pipeline is depicted in Figure 2.
Decoder We employ BART decoder, which takes the output embeddings of the encoders {Z1, ..... Zn, 21, ..., 21.,) and generates sequence h Rm where is the length to generate. The generated hidden states are utilized in two scenar- ios. (i) In the standalone scenario, we apply a linear head W e d to obtain a token sequence Y = hW, where Y e gmxx is the sequence of

---

(b)
Cream
Patchos
Vision Encoder
Selt Houal Prompt
Objjetts
Decoder
WT
Encoder
OCR Boxes
Learned Query
What the number people wearing a scarf and trousers in the left photo?
Question:
Answer:
(a) Overview
There is "one person."
(b) Contrastive Feature Alignment
Wear
Encoder
Vision
Patches
Person
Aligned Fearture
Object Detector
Space
Auxillary Encoder
Objects
phnpiayera.joam
fashionable
OCR
johnplayers
fashionsan nable
Boxes
Figure 2: Overview of Cream's Framework. (a) Image patches fed into the vision encoder, while information extracted from off the-shelf detectors is processed through the auxiliary encoders if available. The encoded vectors are concatenated and then cross-attended query as inputs, serves soft visual prompter for the LLM. Note that the encoders frozen during the training with LLMs. (b) Encoded vector representations are effectively aligned using contrastive learning scheme.
decoder. decoder, receiving both learned query vector
LLM's s input dimension. In both scenarios, language modeling (LM) loss, where the model generates a target sequence of token embeddings conditioned on the image. The details of the train- ing objective will be explained Section 3.3.3.
Auxiliary Encoder
2D Position
X
X
X
Fimbedding
Type Embedding
[OCR]
[OCR]
[OBJ]
[OBJ]
3.1.2
Contrastive Feature Alignment
Text Embedding
'Jeans"
"JOHN"
"Wear"
"Persou"
To enhance the understanding of texts and objects in images, introduce auxiliary encoder along- side the vision encoder (Figure 2). It is critical note that alignment of features from these distinet encoders within a shared space not guaranteed. As observed in our analysis in Section 4, the en- coded information often suffers from misalignment, leading to a model performance degradation. To address this challenge, and simple CL strategy during the model training phase. This strategy employed guarantee the alignment of feature embeddings, which turn, enhances the overall performance the model.
Figure 3: Token embeddings in the auxiliary encoder. The 2D positional embeddings are computed using the center point of each bounding box. Text embeddings are obtained through a lookup operation subword embedding matrix. simplicity, words are plotted instead of subwords.
predicted tokens and is the vocabulary size. (ii) When integrated with an LLM, the hidden states linearly mapped be used as soft visual prompt that combines Cream's visual understanding capa- bilities with the LLM's language processing abil- ities: h' = hU, where e dpdddd and is the
introduce efficient
As shown in Figure 3, we utilize detectors to

---

obtain feature evidence (e.g., texts and objects) and encode their box coordinates and text labels into vector representations. We assume that an image patch can physically overlap with certain boxes, im- plying that the visual patch embedding should en- compass semantically relevant information similar to its corresponding auxiliary information. Given this presumption, we establish the positive relations between visual patch embedding and its corre- sponding embeddings from the auxiliary encoder, while considering all other relationships as negative pairs. Our CL approach differs from conventional image-level CL methods like CLIP (Radford et al., b021b) as our method accommodates more pair- wise relations within an image, leveraging multiple available feature evidence.
Tiger
[TR] prompt: Read texts,
Tipr Anticle "rdinssrediress
redircus
prompt: Read masked largett
image.
Thase
[apt) prompt: Explain the image. Wikipolis antilee
This image appers
bua yn
[QA] prompt: W What the sclentific name of tiger?
Pomihera rigis.
crcunes weth nis lee in serter
[QG] prompt: Make question
of eqcar haure
where the answer Panthera ligris.
Thery aleo den
What is the sciestifk name of tiger?
Figure 4: Unified multitask framework. The list of full prompts that we used is available Appendix A.3.3.
size hidden states that the LLM can utilize. By modifying the attention mechanism to allow bi- directional flow, Cream's decoder effectively ex- tracts visual prompts, enhancing performance in visually-situatee NLU tasks.
It is worth noting that our approach differs from conventional methods that directly input visual em- beddings and OCR tokens to LLMs, e.g., BLIP-2 and InstructBLIP (Li et al., 2023; Dai et al., 2023), Our method extracts relevant information from the input image and utilizes Cream's decoder ag- gregate the extracted information. The generated visual prompts therefore contain OCR information, while reducing the computational overhead.
For CL, tron (MLP) fg : Id - Rdd where d* is hy- perparameter for a dimension of a common space, Most settings are similar to those of Khosla al. (2020). More specific details are provided in Ap- pendix 1.4.1. The CL objective can be expressed as follows:
2-layer Multi-Layer Percep-
28(Vi, Vÿ)
Σ log (am)~Pa
() s(vi, v&) + v(V)
3.3 Model Training
3.3.1 Tasks
where the sets [v:|V; (z,,... 1,
W
Text Reading (TR) Cream reads text from top to bottom within images (Kim et al., 2022). In this task, the output of auxiliary encoder is masked to support the model learning a text reading ability.
< I} and (v,lv, (ž). (2,...,,a))
&
j < 2/} are uniformly sampled from all pos- itive pairs in (z,...., ...,, Zn; 21..... 3 We refer A(i,j) as
The denominator accumulates the similarity scores over all neg- ative pairs. The function s(x,y) is defined where
Masked Text Prediction (MTP) Cream predicts obscured characters in randomly masked OCR boxes. This task can be interpreted as an expan- sion of masked LM (Tay et al., 2022) to the visual domain.
y(x,y) cos(fo(x), denotes computing the cosine similarity between its vector inputs, with MLP pa- rameterized by f. The r is the temperature param- eter that modulates the softmax sharpness. The proposed CL encourages the aligmment of embed- dings from both encoders in the feature space, re- sulting in performance improvement. We validate the effectiveness of CL in our analyses ($ 4).
Captioning
Cream generates captions encapsu- lating scene and object details, enhancing image- content understanding and object recognition.
Question Answering (QA) Cream answers ques- tions using language-iman imagg comprehension vi- sual contexts, emphasizing relevant image regions and textual information.
3.2 Integration of Cream and LLMs
The integration method is built upon the work pro- posed in BLIP-2 (Li al., 2023), where Cream's 's decoder generates visual prompts for the LLM to generate text responses. We adopt the learned query mechanism from BLIP-2, which uses train- able embeddings as inputs for the Cream de- coder. This allows for the generation of fixed-
Question Generation (QG) QG prompts Cream to create question sentences for provided answer texts by swapping QA components.
3.3.2 Unified Multitask Framework
The tasks of text reading, MTP, captioning, QA, and QG are interrelated and can be addressed using

---

similar approaches. They involve extracting a text sequence based on task-specific queries given an input image and its features. Our unified training framework (See Figure 4) takes prompts and im- ages as input and generates desired answer texts for all tasks. Unlike other document understanding methods that use single task-specific prompts (Kim et al., 2022; Tang et al., 2022), Cream is trained with natural language-based prompts, facilitating seamless integration into LLMs. Our prompt is distinet to other methods such as Donut (Kim et al., 2022) and UDOP (Tang et al., 2022), which employ task-specific prompts.
Q1: What the Establismmnt?
A1: 1861
02: What "Time" uader "Greek" the ?
AZ: 6:15 pm (CWI
What "PPatte" under "clere, elerer" in the
Q4: What other albumss?
AG: Muison du M-flo
QS: What "Nominee" in the card.?
ASi Koben Colvan Macauley
Figure 5: Examples of synthetic VQA datasets. Ex- amples of other datasets are available in Appendix A.6.
InfographieVQS
ChariQA
DocVQA
3.3.3 Objective
Inspired by modern mee-rrining-and--ne--mmm strategies (Kim et al., 2022) and curriculum learn- ing strategies (Soviany et al., 2022), we gradually increase the proportion of supervised QA data dur- ing training, initially focusing on text reading and image captioning. Two main objectives are used during training: LM and CL. loss. The LM objec- tive is to minimize cross-entropy loss between predicted and ground truth token sequences. In line with Vaswani al. (2017), the teacher-foreing scheme (Williams and Zipser, 1989) is employed, using ground truth as input during training for ac- curate contextual learning. The CL. objective en- courages alignment of embeddings in the feature space between vision and auxiliary encoders. The CL objective is defined in Equation 1.
Figure 6: Evaluation benchmarks. We evaluate mod- els on visual context-rich Document VQA benchmarks.
decoder modules are initialized with mBART (Liu et al., 2020). We test two model sizes: the main (18, 12, 12 layers for vision, auxiliary encoders. and decoder, respectively, with a 14x 14 patch size) and a smaller ablation model (9, 6, 6 layers with a 32x 32 patch size). For LLM integration tests, we use Vicuna7B (Chiang et al., 2023).
Datasets Table I provides an overview of train- ing datasets and their statistics. For TR and MTP, employ IIT-CDIP (Lewis et al., 2006) and WE- VICOB (Kim et al., 2023). WEBVICOB is visual corpus generator for Wikipedia dump. We create a visual corpus of size 30M. For Captioning, we use CC3M dataset, which contains general im- ages and accompanying text deseriptions. For QA and QG, we introduce various supervised VQA datasets to improve Cream's 's visual understanding capabilities. To further boost the text-rich image understanding, using Wikipedia data source, we generate three synthetic VQA datasets: WKVVQA, SquadVQA, and TydiVQA. WKVVQA comprises synthetic document images with key-value pairs ex- tracted from the Wikipedia. Samples of WKVVQA are shown in Figure 5. Both SquadVQA and Tydi- VQA expand unimodal datasets (Rajpurkar et al., 2018; Clark et al., 2020) by rendering its context page with WEB VICOB. More details on the dataset construction are available in Appendix A.5.
The losses are combined using a weighted sum, LL XLCL. where À is the scale factor of the CL loss. This training objective ensures effective alignment of encoded information and high perfor- mance in eosually-sitaatee NLU tasks, as demon- strated in our experiments ($ 4). When training Cream integrated with LLMs, we freeze the LLM and Cream's 's encoders while updating the Cream decoder via gradient descent-based training. Note that, in this phase, only the LIM from the LLM's output layer is active.
4 Experiments and Analyses
4.1 Setups
This section provides major details on experiments. More details are discussed in Appendix A.4.
Model Configurations The vision encoder is ini- tialized with LAION (Schuhmann et al., 2022) 2B pre-trained OpenCLIP (Radford et al., 2021b; II- harco et al., 2021). The auxiliary encoder and the
hhttps://gitee tww.::b..tttut...e

---

Datasel
Task
Size (H1mg)
Phase
Task Propuction
IIT-CDIP (Lewis al., 2006)
/
Stundalone
TR. Cape., QA, QG (22, 22, (100%)
WERVICOB (Kim 2023)
/ MTP
30M
LLM Integration
QA (100%)
CC3M (Sharma al., 2018)
Captioning
Table 2: Task proportions according to phases.
CharQQ (Masry ot al., 2022)
QA/O /QG
18K (train)
InfoVQA (Mathew et al., 2022)
QA/Q0
4K (train)
DocVQA (Tito et al., 2021)
QA/QG
11K (train+val)
VisualMRC (Tanaka et al., 2021)
QA/QG
9K (train+val)
class label texts. More details on the detectors and label texts can be found in Appendix A.2.
DVQA (Kafle ct al., 2018)
QA/QG
200K (train)
OCRVQA (Mishra et al., 2019)
QA/QG
146K (train)
STVQA (Biten et al., 2019)
QA/QG
17K (train)
Environment and Hyperparameters We sum- marize task proportions during training in Table 2. The training starts with batch size of 384, fixed learning rate of le-4, for 220K steps. Next, ad- just the batch proportion and hyperparameters for another 275K steps; a batch size of 96, and a learn- ing rate of 5e-5 with a decaying scheduling. The LLM integration employs a batch size of 192, 50K steps with 0.5 GPU days with 32 A 100 GPUs, and a cosine-scheduled learning rate of le-4. Other major hyperparameters are (A, T, d, d k. d', n, n. I) (0.5, 0.07, 1024, 128, 224, 4096, 3072, 300,90)
TexVVQA (Singh al., 2019)
QA/QG
25K (train+val)
VizWizVQA (Gurari et al., 2018)
QA/QG
15K (train)
VQAv2 (Goyal et al., 2017)
QA/QG
83K (train)
WTQ (Pasupat and Liang, 2015)
QA/QG
14K (train)
SquadVQA
QA/QG
130K (train)
TydiQA
QA /QG
4K (train)
WKYVQA
QA/QG
800K
Table I: Statistics of the training datasets.
Evaluation The models are evaluated on text- rich VQA benchmarks, including ChartQA (Masry et al., 2022), InfographicvQA (InfoVQA) (Mathew et al., 2022), and Doc (Tito et al., 2021) (Fig- ure 6). While DocVQA serves a representa- tive text-rich benchmark, its majority of extrac- tive QA samples demand less reasoning capabil- ity compared the other two datasets. Both In- foVQA and ChartQA pose significant challenges, with ChartQA requiring advanced reasoning skills, exemplified by GPT-4's (OpenAT, 2023) chain-of- thought approach specifically tailored for it. Fur- thermore, Info necessitates a thorough under- standing of large images' content. Doc and InfoVQA are evaluated via the official competition leaderboard with confidential test sets, complying with recent VDU literatures like Donut (Kim et al., 2022) and Pix2Struct (Lee et al., 2022). ChartQA has a public test set and is evaluated with an exact- match-based accuracy, as conducted by previous literatures (Lee et al., 2022; Masry et al., 2022). Throughout the evaluation process, we assess all models under the real-world scenario, meaning that we do not utilize ground truth OCR during the testing phase. Instead, we rely on off-the-sheef detectors, which may contain some errors.
4,2 Results
Table 3 presents the performance of various Frozen LLM integration models on text-rich VQA bench- marks. Our proposed Cream integration demon- strates significant improvements compared to other Frozen LLM integrations, particularly in bench- marks that require advanced visual understanding capabilities. A notable characteristic of Cream in- tegration is its fixed-size soft visual prompt usage, which remains constant at 224 tokens regardless of the number of texts within the image. This afficiency-orientee approach contrasts with meth- ods that input all OCR tokens into the LLM. Con- sequently, our model does not rely on exceedingly large token lengths (denoted as |OCR)) to pro- cess document information, thereby increasing effi- ciency. Figure 7 illustrates the non-negligible size of OOCR), which leads inefficiencies in conven- tional techniques. Table 3 demonstrates Cream's superior performance over existing LLM integra- tion methods, even in the absence of off-the-shelf detectors. In this setting, the inference speed was 0.25 sec/sample. In contrast, OCR-Vicuna78 (ex- cluding OCR time) had speed of 0.5 sec/sample. Further details can be found in Appendix A.2.
Off-the-Shelf Detectors For OCR, we employ CLOVA OCR API', while for general object de- tection, we utilize OWL-ViT* from Minderer et al. (2022). The MS-COCO dataset (Lin et al., 2014) supplies the 80 class labels required for semantic
Table 4 presents standalone performance. The first group consists of state-of-the-art VDU mod- els fine-tuned on each benchmark, with ChartQA's UDOP score obtained using official implemen- tation and training settings (more details in Ap- pendix A.4.3). The second group shows multi-task
hhttps://rrc.ww cwt.cccccccuub.... us hhttps://coova.ail ailorr//n http://gggiggaaccccoooo///o///// google/ lanit-llarg--atchh

---

Model
Prompt Length
Use Auxiliary
CharlQA
InfoVQA
DocVQA
LLaVA-Vicuua78 (liu et al., 2023a)
0.5
AlaVA-Vicunal.3t (Liu et al., 2023a)
256
1.4
3.1
5.9
BLIP2-OPT6.77 (Li et al., 2023)
4.6
11.0
3.7
1-1P2- 1B (l.i et al., 2023)
32
4.4
11.4
8.6
Cream- Vicuma7B w/o off-the-shelf detectors
22.1
41.1
OCR-Vicuma7B (Chiang al., 2023) OCR- 13R--Viuaa (Chiang et al., 2023)
JOCR
6.2
13.6
29.2
JOCR
3.7
23.7
31.4
OCR-GPT3.5
[OCR
v
15.9
26.6
62.4
OCR-GPT4 (OpenAl, 2023)
JOCR
34.3
25.0
75.9
BLIP2XOCR-OPT6.7 7B (Li et al., 2023)
32+|OCR
17.5
30.4
6.2
BLIP2XOCR--PanTS-. (Li et al., 2023)
32+|OCR
18.6
36.6
63.8
Cream- Vicumai (Proposed)
224
63.0
43.5
79.5
Table 3: Experimental results for various models on visually-situater NLU tasks. The term |OCR| indicates
the number of tokens required to input OCR texts to the models. Cream, when integrated with the frozen Vicuna, significantly outperforms other LLM integrations with an efficient prompt length.
Viama-ih
80
Requency
12
60
20
S9n
1530 Number of OCR OCR colorn
2016
30% OCR Drop Rate
Figure 7: Visualization of LLM token consumption induced by OCR. We DocVQA dataset (Tito al., 2021) and tokenizers of Vicuna (Chiang et al., 2023) and F5anT5 (Chung et al., 2022).
Figure 8: Robustness in OCR deprivation. We test model robustness by gradually increasing OCR drop rate on Doe samples. Cream shows robust performance even when no auxiliary information is given.
Model
Size
Chart
Info
Doc
4.3 Analyses
Single-task Finetuned Sfate-of-the-art
(Raffel et al., 2020)
0.8B
36.7
70.4
We show some key findings in this section. Addi- tional details and results are Appendix A.I.
Donut (Kim al., 2022)
0.2B
41.8
21.7
67.5
MatCha (Liu et al., 2022)
0.3B
64.2
37.2
74.2
Pix2Struct (lee et al., 2022)
1.3B
58.6
40.0
76.6
UDOP (Tang 2022)
0.7B
60.7
47.4
84.7
Visual Prompt Length Cream generates high- quality fixed-size visual prompts, reducing the com- putational cost associated with integrating OCR to- kens into LLMs. The attention complexity per layer is (n²d), with and d denoting the sequence length of tokens and the size of hidden dimension, respectively. Reducing the input token length (ñ) significantly decreases the complexity LLMs. Figure 7 showcases the substantial token consump- tion when incorporating OCR directly into LLMs. This underlines the potential computational advan- tages of Cream for visually-situated NLU tasks compared to other LLM integration strategies.
Connolled Multi-task Model
UDOP
0.7B
60.2
43.8
77.3
Cream
0.6B
62.7
41.0
81.2
Table 4: Standalone performance. Cream shows com- parable results recent VDU state-of-theearts.
models, with Cream showing comparable perfor- mance to state-of-the-art models even when consid- ering its multi-task setting and use of off-the-shelf detection results. Compared to the LLM integra- tion, the standalone shows lower scores on chal- lenging benchmarks, but higher scores in DocVQA. The LLM integration excels questions requiring reasoning but struggles with those benefiting from direct image inspection. Including further analyses on the benefits LLM integration, we discuss the proposed Cream's ' effectiveness and its robustness to OCR in the following analysis section.
Robustness on Auxiliary Information Figure 8 illustrates Cream's high robustness against missing OCR in the text-rich VQA benchmark. We ob- served that Cream remained effective even without any OCR box input. Furthermore, CL significantly contributed to the increased robustness. We also

---

Feature Representations wio CL
DocVQA
InfoVQA
Model
Form
Table / List
Map
Arithmetic
taxt
mm
leti
In
Cream (Standalone)
88.7
80.9
38.3
31.7
Cream Vicuma7B
86.8
78.1
30.9
37.9
hubition
Director
Eirecter- teot
Table 5: Comparative analysis between L.LM Inte- gration and Standalone. The LLM Integration shows superior performance in tackling arithmetic problems,
13
Feature Representations w: CL
Img
kry
text
seat
Nedcne
Model
Acc.f
ANLS;
nED1
BERT!
Nedcaw
BLIP2&OCR LLIP2-0CR-FluTT--LBB
18.6
24.7
67.1
76.3
24.5
Civan- VicunaBB
63.0
60.3
314
91.0
2.0
Table 6: Evaluation results with multiple metries. The models are assessed with the ChartQA benchmark using Accuracy, ANLS, nED, BERTScore, and PPL..
Figure Visualization the common feature space through PCA. The proposed CL aids in aligning fea- tures which possess similar semantics.
tested other publicly available OCR APIs/engines, Using lightweight CPU-based OCR3, Cream ex- hibited a smaller performance drop (-19% compared to UDOP (-29% demonstrating Cream's s superior robustness. More detailed analy- ses are provided in Appendix A.1.1.
Cream model. Contrariwise, although most met- rics underscore subpar performance for BLIP2, we find that its responses are not outrightly branded as nonsensical by the result of BERTScore.
Feature Space Visualization To further under- stand the role of CL., we conduct a visualization analysis the common feature space. Figure 9 presents PCA results for the common feature space generated by the two encoders. It is evident that the CL-applied space more effectively removes the modality gap when excluding the first component. Employing the 2nd and 3rd components, we ob- served enhanced alignment and better clustering in the CL-applied space. These observations sug- gest that our CL leads to better-aligned embeddings from both encoders, significantly contributing performance improvement. Further analyses be found in Appendix A.1.2.
Efficacy Analysis of LLM Integration Table shows comparative results between the LLM Inte- gration and Standalone models. The assessment is concentrated some categories which displayed conspicuous and statistically significant disparities in scores. The table clarifies that the Standalone Cream exhibits superior performance in scenarios that necessitate elementary key-value identification without the need for intricate reasoning, or tasks that involve comprehending extensive large map figures within an image. In contrast, the LLM In- tegration demonstrates higher competency in arith- metic problems that necessitate logical deduction.
Evaluations with Diverse Metries for VQA For a comprehensive analysis of model performance, we utilize the test set of ChartQA benchmark and evaluate models using a series of diverse metrics. These encompass Average Normalized Levenshtein Similarity (ANLS) (Tito et al., 2021), Normalized Edit Distance (nED), BERTScore (Zhang et al., 2020), and Perplexity (PPL.). This suite of metrics provides well-rounded enhancement to the con- ventional exact-match accuracy, shedding light on various facets of model capabilities. The evaluation results are concisely summarized in Table 6. No- tably, findings from ANLS and nED investigations depict smaller performance gap than accuracy, yet unequivocally uphold the preeminence of the
5 Conclusion
In this paper, present Cream, novel approach overcoming the constraints of current LVLMs for visual tasks on text-rich images. Cream's robust architecture synergizes a vision encoder, auxiliary encoder, and sophisticated techniques, including contrastive feature alignment. Our comprehensive evaluations confirm Cream's 's promising language- image understanding capabilities and robustness against OCR errors. The integration of Cream with LLMs provides solid foundation for future im- provements in visually-situated language compre- hension. We believe our findings can easily be extended to other domains/tasks regarding visually- situated natural language understanding.
hhttps: httpp///w.oomaaaaeeeaaeleeeeeeee

---

Acknowledgements
Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkkowsi,, Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, and Karén Si- monyan. 2022. Flamingo: a Visual Language Model for Tew-Shot Learning. In Advances in Neural Infor- mation Processing Systems, volume 35, pages 23716- 23736. Curran Associates, Inc.
The authors especially thank Seung Ho Choi, Jin- bae Im, and members of NAVER Cloud Hyperscale AI Vision Understanding Team for helpful discus- sions and encouragement.
Limitations
In this study, we have primarily focused single- page image processing and successfully established a pioneering framework for integrating Cream with LLMs to address text-rich visual document un- derstanding tasks. However, certain challenges and complexities associated with multi-page image analysis remain unexplored. Given the increasing demand for handling multiple images simultane- ously, particularly in applications such as chatbot- like UIs where LLMs are commonly employed, extending our approach to multi-page processing represents crucial aspect and calls for future re- search. Overcoming this limitation could involve distinct considcrations, such as developing visual instruction data specifically tailored for multi-page images.
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. 2021. DocFormer: End-to-FEnd Transformer
Document Understand- ing. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 973-983, IEEE
Ali Furkan Biten, Ruben Tito, Andrés Mafla, Lluís Gómez i Bigorda, Marçal Rusiñll, C. Jawa- har, Ernest Valveny, and Dimosthenis Karatzas. 2019. Scene Text Visual Question Answering. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Novemeer 2, 2019, pages 4290-439000 030004400..00 IEEE
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever. and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neurul Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual.
Ethics Consideration
In our work, we inherit the ethical concerns of ex- isting large-scale language models, such as data bi- ases and privacy considerations. To mitigate these issues, advocate for striet protocols during pre- training data curation, especially in public appli- cations. Our model's pre-training uses controlled public data sources. Privacy-sensitive document processing, e.g., identification cards, requires dili- gent data handling practices for LLM development. Excluding such samples from training datasets is essential to prevent potential privacy breaches and unintended consequences. While our current ap- proach relies the autoregressive decoder's 's di- rect output, eliminating complex post-processing, it may be worth considering the investigation of post-processing techniques that address biases and privacy issues. This could provide an added layer of protection and ensure that model outputs adhere to the ethical guidelines within the field.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing, 2023. Vicuna: An open- source chathot impressing gpt-4 with 90%* chatgpt quality.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan. Hyeontaek Lim. Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira,
References
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan

---

Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellsr-Hllsteem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways.
in Visual Question Answering. 2017 IEEE Con- ference on Computer Vision and Pattern Recognition. CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1325-6334, IEEE Computer Society.
Danna Gurari, Qing L., Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answer- ing Visual Questions From Blind People. In 2018 IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages (608--617... IEEE Computer Society.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valtrr, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Zhao. Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling Instruction-Fineen Finetuned Language Models.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recog- nition. In 2016 IEEF Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Com- puter Society.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for infformationnseeking question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguisticx, 8:454-470.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training Compute-Opiiml Large Language Models.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning. ArXiv preprint, abs/2305.06500..
Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. 2022. BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extrac- tion from Documents. In Thirty-Sixtt AAAI Confer- ence on Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applications of Ar- rificial Intelligence, IAAI 2022. The Twelveth Sym- posium Ealucationa Advunces in Artificial In- telligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 10767-10775. AAAI Press.
Brian Davis, Bryan Morse, Brian Price, Chris Tens- meyer. Curtis Wigington, and Vlad Morariu. 2023. End-to-end Document Recognition and Understand- ing with Dessurt. In Computer Vision ECCV 2022 Workshops, pages 280-296, Cham. Springer Nature Switzerland.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x 16 Words: Transformers for Image Recog- nition at Scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event. Austria, May 3-7, 2021. OpenReview.net.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. LayoutLMv3: Pre-Training for Doc- ument Al with Unified Text and Image Masking. In Proceedingy of the 30th ACM International Confer- ence on Multimedia, MM '22, page 4083-4091, New York, NY, USA. Association for Computing Machin- ery.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Ser- manet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023. M-LM--:: An Embodied Multimodal Language Model.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Lud- wig Schmidt. 2021. OpenCLIP.
Kushal Kafle, Brian L. Price, Scott Cohen, and Christo- pher Kanan. 2018, DVQA: Understanding Data Vi- sualizations via Question Answering. 2018 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5648-5656. IEEE Computer So- ciety.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA Matter: Elevating the Role of Image Understanding

---

Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1769-6781, Online. Association for Computational Linguistics.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision ECCV 2014, pages 740-755, Cham. Springer Inter- national Publishing.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022. MatCha: Enhancing Visual Language Pretrain- ing with Math Reasoning and Chart Derendering. ArXiv preprint, abs/2212.09662.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yongiong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Super- vised Contrastive Learning. In Advances in Nearal Information Processing Systems 33: Anmual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jac Lee. 2023a. Visual Instruction Tuning.
Jihyung Kil, Soravit Changpinyo, Xi Chen, Hexiang Hu, Sebastian Goodman, Wei-Lun Chao, and Radu Soricut. 2022. PreSTU: Pre-Training for Scene-Text Understanding.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- rics, 8:726-742.
Donghyun Kim, Teakgyu Hong, Moonbin Yim. Yoon- sik Kim, and Geewook Kim. 2023. On Web-based Visual Corpus Construction for Visual Document Un- derstanding. In Document Analysis and Recognition- ICDAR 2023, pages 297-313, Cham. Springer Nature Switzerland.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023b. On the Hidden Mystery of OCR in Large Multimodal Models. ArXiv preprint, abs/2305.07895.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam. Jinyoung Park. Jinyeong Yim, Won- scok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. OCR-Free Document Un- derstanding Transformer. In Computer Vision ECCV 2022. pages 498-517, Cham. Springer Nature Switzerland.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021, Swin Transformer: Hicrarchical Vision Transformer using Shifted Windows. In 2021 IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9992- 10002. IEEE.
Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandel- wal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2022. Pix2Struct: Screenshot Parsing Pretraining for Visual Language Uderstanding.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty. and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263- 2279, Dublin, Ireland. Association for Computational Linguistics.
Lewis, G. Agam, Argamon, O. Frieder, D. Gross- man, and J. Heard. 2006. Building a Test Collection for Complex Document Information Processing. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, page 665-666, New York, NY, USA. Association for Computing Machinery.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. 2022. InfographicVQA.. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion (WACV), pages 1697-1706.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-seet ceaecce-to-equueee to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 7871-7880 Online. Association for Computa- tional Linguistics.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy. Aravindh Mahendran, Anurag Arnab. Mostafa Dehghani, Zhuoran Shen, Xiao Wang. Xi- aohua Zhai, Thomas Kipf, and Neil Houlsby. 2022. Simple Open- Vocabulay Object Detection. In Com- puter Vision ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part X. page 728-755, Berlin, Heidelberg. Springer-Verlas Verlag.
Junnan I.,, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Imag Pre- training with Frozen Image Encoders and Large Lan- guage Models.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. OCR-VQA: Visual

---

Question Answering by Reading Text in Images. In 2019 International Conference on Document Analy- sis and Recognition (ICDAR), pages 947-952.
of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1-140:0
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't t know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume 2: Short Papers), pages 784-789. Melbourne, Australia. Association for Computational Linguistics.
OpenAL. 2023. GPT-4 Technical Report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa R Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk. and Jenia Jitsev. 2022. LAION-5b An open large-scale dataset for training next generation image-ter text models. In Thirty- sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track.
Panupong Pasupat and Percy Liang. 2015, Composi- tional semantic parsing on semi- structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 1470- 1480, Beijing, China. Association for Computational Linguistics.
Sakib Shahriar and Kadhim Hayawi. 2023. Let's have a chat! A Conversation with ChatGPT: Technology. Applications, and Limitations.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hyperyyeed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Anmual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers). pages 2556-2565, Melbourne, Australia. Association for Computational Linguistics.
Rafal Powalski, Fukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michal Pietruszka, and Gabriela Palka. 2021. Going Full-TILT Boogie on Document Understanding with Text-Image-Layone Transformer. In Document Analysis and Recognition ICDAR 2021, pages 732-747, Cham. Springer International Publishing.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021a. Learn ing Transferable Visual Models From Natural Lan- guage Supervision. In Proceedings of the 38th In- lernational Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA Models That Can Read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20. 2019, pages 8317-8326 Com- puter Vision Foundation/ / IEEE.
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum Learning: A Survey. Int. J. Comput. Vision 156-61555
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021b. Learn- ing Transferable Visual Models From Natural Lan- guage Supervision. In Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763, PMLR
Mareus Spruill. 2007. Asymptotic Distribution of Co- ordinates on High Dimensional Spheres. Electronic Communications in Probability, 12(none):234 247.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. 2021. VRs Machine Reading Comprehension on Document Images. In Thirty-Fifth AAAI Conference Artificial Intelligecee AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, TAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9. 2021, pages 13878-13888, AAAI Press.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, al. 2021. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. ArXiv preprint, 1t./22122.11446
Zineng Tang. Ziyi Yang, Guoxin Wang, Yuwei Fang. Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2022. Unifying Vision, Text, and Layout for Universal Document Processing. ArXiv preprint, abs/22 12002233..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits

---

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Dara Bahri, Tal Schuster, Huaixiu Zheng, Neil Houlsby, and Donald Metzler, 2022. Unity- ing Language Learning Paradigms. ArXiv preprint. abs/2205.05131
Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou, 2021. Lvv: Multi-modal pre- pre-training for visually-rich document understanding. In Proceed- ings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 lik International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2579-2591, Online. Association for Computational Linguistics.
Rubèn Tito, Minesh Mathew, C. V. Jawahar, Ernest Val- veny, and Dimosthenis Karatzas. 2021. ICDAR 2021 Competition on Document Visual Question Answer- ing. In Document Analysis and Recooniiionn-CLUI 2021, pages 635-649, Cham. Springer International Publishing.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. LayoutLM: Pre-training of Text and Layout for Document Image Understand- ing. In KDD '20: The 26th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, Virtual Event, CA. USA, August 23-27, 2020, pages 1192-1200. ACM.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mPLUG-OWI Modularization Empowers Large Language Models with Multimodality. ArXiv preprint, abs/2304. 14178.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill, 2021. Multimodal Few-Shot Learning with Frozen Lan- guage Models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 200-212.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre- trained Transformer Language Models.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. OFA: Unifying Ar- chitectures, Tasks, and Modalities Through Simple Sequence-to-SSqueeec Learning Framework. In In- ternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23318-23340. PMLR.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhane- ing Vingon-Language Understanding with Advanced Large Language Models.
A
Appendix
A.1
Additional Analysis Results
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck. Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. 2022b. Image as Foreign Lan- guage: BEiT Pretraining for All Vision and Vision- Language Tasks.
A.1.1
Performance Dependency of OCR Engines
In the pursuit of evaluating the models in realistic contexts, we present a set of supplementary ex- periments conducted using outputs from diverse, readily available OCR engines to assess model de- pendency on OCR. As demonstrated in the main manuscript, our proposed approach, Cream, dis- played superior robustness when confronted with subpar OCR results. In this subsection, we offer further analytical outcomes produced by incorpo- rating other prolific, public-access OCR engines. We not only tested CLOVA OCR AP1 but also
Wenjin Wang, Yunhao Li, Yixin Ou, and Yin Zhang. 2023. Layout and Task Aware Instruction Prompt for Zero-shot Document Image Question Answering. ArXiv preprint, abs/2306.00526.
Ronald J Williams and David Zipser. 1989. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural computation, 1(2):270- 280.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha
hhttps://clov..a//orrr/

---

50
91.2
Creon UDOP
Cosine Similarity Histogram
Cosine Suilarrt Hiltorram wio
-10%5
70-
-19%
Denaty
Denaty
50 Clova
EASY Usod OCR Engine
Paddle
-1.
-0.5
0-0 Simileity
3.5
1.5
-3.3
-3.5
0.0
10
Simlirity
Figure 10: Comparison of Cream and UDOP ro- bustness across multiple OCR engines. The perfor- mance Cream and UDOP assessed on VQA using CLOVA OCR, EasyOCR, and PaddleOCR gines. Cream displays better resilience, maintaining performance despite the varying capabilities of OCR engines.
Figure 12: Cosine similarity histograms. The his-
togram of CL exhibits distribution that closely resem- bles a Gaussian distribution.
Model
ChartQA
DocVQA
InfoVQA
Creamsual
56.7
68.5
30.7
wf PaddleOCR at rest
53.1 (46.4%)
51.9 (-24 3%)
25.5 (-16.9%)
disubll ouc, at test
41.1 (-27.5%)
40.0 (-41.7%)
(-5.6%))))
70-
65.5
06.1
VETD
Chbiliut
Cletiiiesa CL
Creaminnt
54.6
wf PoddleOCC at test
47.3 (-13,39))
38.5 (-41.9%)
25.1 (-19.8%)
60
-14%
disoble out at tear
10.6 (-80.6%)
8.3 (-87.4%)
13.1 (-58.1%)
-24%
50-
VETD
47.8
49.2
15.5
4.%
Table 7: Ablation study results for Creamsmall- The table shows that CL mitigates performance degradation due to absence, or noise of auxiliary information (comparing blue values vs. red values). Note that Pad- dleOCR, as CPU- based OCR engine, is lightweight has lower performance.
30 Clova
EASY Usod OCR Engine
Puddie
Figure 11: Assessing CL's impact on robustness across multiple OCR engines. We conduct evaluations on the DocVQA using CLOVA, EasyOCR, and Pad- dleOCR engines. The figure demonstrates that training with CL enhances robustness and mitigates performance degradation, even with varying OCR engine capabili- ties.
VETD, it suggests that the addition of the auxil- iary encoder may not be justified, As indicated in the figure, by incorporating the proposed CL technique, the overall enhancement performance is observed, enabling Creamsmall to outperform the baseline VETD, even with the deployment of CPU-based lightweight OCR engine- Pad- dleOCR. Table 7 presents a detailed ablation study for Creamsmall- The robustness trend of CL against the noise or absence of auxiliary information be observed across the entire evaluation datasets.
evaluated EasyOCR? and PaddlcOCRS. Note that, PaddleOCR operates as a lightweight, CPU-based OCR engine, thus potentially acting as a preferred option for those seeking to decrease overall model deployment costs.
Figure 10 presents comparative performance analysis of Cream and UDOP the text-rich Doc- ument VQA benchmark, VQA. As can be ob- served in the figure, Cream exemplifies superior robustness. Figure 11, additionally, exhibits ab- lated models of Cream, evaluating the influence CL. These findings are aligned with the trends noted in the main manuscript, underscoring that our implementation of CL improves robustness to OCR errors.
A.1.2 Additional Analysis on the Common Feature Space with CL
Figure 12 illustrates the histograms of cosine sim- ilarity by calculating the cosine similarity be- tween two randomly chosen embeddings within the shared common space. Under the assumption that the embedding space contains random (unit) vectors, Gaussian distribution is expected for the histograms (Spruill, 2007). The red line depicted in the figures corresponds to the Gaussian distri- bution that minimizes KL divergence for each his- togram distribution, specifically (N 1/140) for the CL. condition and (0, 1/3) for the non-CL. condition. These results suggest that, with the inte- gration of CL, the embeddings are distributed more
Figure 11 shows the performance of new ab- lated model, Vision Encoder and Text Decoder (VETD), which derived from Creamsmall by removing the auxiliary encoder in the architec- ture. We use VETD as performance baseline: if Cream's performance does not exceed that of
'https: htttts::/github... httpgttuuu..om//aidecc/////aa// httpl//://gggiiuu..o///aa/aaddde

---

randomly or broadly across the embedding dimen- sion, factor that may significantly contribute to the enhanced performance of the Cream model.
Model
P1
P2
P3
P4
P5
CCR--Vicna7B
25.6
19.2
20.1
6.4
28.3
DCR-Vicunal 38
28.2
24.8
29.5
7.5
29.0
OCR-GPT3.5
50.1
60.4
47.9
17.9
60.5
A.1.3 Analysis of Working Examples
OCR-GPT4
52.1
63.8
60.2
30.9
70.4
Figures 13, 14, and 15 depict several working instances. Overall, the LLM integration model demonstrates a strong edge in tasks requiring arith- metic reasoning prior knowledge. For instance, the final sample in Figure 13 and the first two sam- ples in Figure 14 necessitate certain basic arith- metic computations. As also showcased with the quantitative evaluation in the main manuseript, the LLM integration model shows its capability for performing numerical operations among elements plotted in infographics and charts.
Table Experimental results for OCR-LLMs DocVQA with different prompt types. The evalu- ation metric is ANLS, assessed after 500 samples from the validation set.
OWL-ViT Large and 0.22 seconds per sample with the OWL- ViT Our experiments revealed lit- tle performance difference between OWL-ViTi.m and OWL-ViTBRse suggesting that more efficient detectors could be tested. When examining the Cream standalone with only the object detector un- available (OCR remained available), minor per- formance drop in DocVQA was observed (from 81.2 1.2 80.9).
Conversely, the standalone model manifests cer- tain advantages when the task demands basic text reading ability from text-heavy documents. We hypothesize this is due the standalone model directly referencing the input image, whereas the integration model must encode the information into the soft visual prompts (vectors). For instance, the instances Figure 15 reveal that the LLM integration model either omits some letters or gen- erates some characters inaccurately, particularly as the image's ' text size shrinks becomes denser.
As highlighted above, off-the-shiee detectors can significantly influence a system's 's inference speed and deployment cost. Bearing this in mind, we designed our model to offer users the flexibility to choose options based on their specific needs. Factoring in all potential configurations, from dis- abling all off-the-shelf detectors using each fea- ture (without parallelism), our model can achieve inference speeds ranging from 0.25 to 4.91 (0.25 0.66 + 4) seconds per sample. Given these degrees of freedom, we anticipate that practitioners will be enabled to construct highly efficient systems using Cream, potentially with further enhancements such as parallelization.
A.2 Details on Off-the-Sheff Detectors and Inference Speed
Our preliminary studies on off-the-shed OCR en- gines, as shown in Figure 10, led us to adopt CLOVA OCR APP for the experiments. On av- erage, processing a sample from DocVQA using the above APT took around seconds. It's worth noting that, Doc often includes text-rich doc- uments, leading to relatively larger API time cost compared to other benchmarks. Although alterna- tive lightweight OCR solutions like PaddleOCR 10 offered faster processing times, they revealed a ticeable quality gap, resulting in many text boxes being missed.
A.3
Considerations on LLM Prompts
A.3.1
OCR-LLM Prompt Variations
We empirically observed LLMs' sensitivity to nat- ural language prompts during testing. In order evaluate the influence of various prompts on overall performance, we conducted a comparative analysis of five different types prompts, all of which are evaluated through the text-rich DocVQA bench- mark (Tito et al., 2021). 1). As shown in Table 8, the choice of prompt significantly influences the over- all performance.
For general object detection, we employed OWL- ViT!l from Minderer et al. (2022), using the MS- COCO 80 class labels as the semantic class label texts for object detection. On average, the API took 0.66 seconds per sample for processing with
Table 9 shows that integrating more specific ditions into the prompts generally results in im- proved performance. More specifically, we ob- served that text extraction tasks yield better out- comes when conditions specifically stipulate that responses should be derived from OCR tokens. Fur-
"https://gittub. http:///////iiuuu..o///aaaaaaada wwtp///aadleaadle//aaaaaaaddeeda
"I "nttps://hugginggaace co.ggogle///
owlvit--large-patch14
"https://gist" http:/ggsttttttt.... git com/ AruniRC/

---

What second leading causse death state prisonera 20187
than other two?
What Documentary?
Maxico
Heart dica360
D0.76
Mesico
Heart
GO
Creams LLM
Creams + LLN
Creare LLM
Figure 13: Working examples on ChartQA benchmark. Both models can handle various types of queries on chart images, but I.L demonstrates an advantage in queries that involve arithmetic reasoning.
Right otraggling
BYTESSIZED
M4ric
28%
200,000
UAE'S FIRST 120 FESTIVAL OF CRICKET
world
What devicse. circle,
and International atar
makinn loxs?
X
X
22
17%
drde
18
13%
Aquaae
Cream LUN
Creamt LLM
Dream LLM
Figure 14: Working examples on Infographic benchmark. Examples that require arithmetic reasoming or prior knowledge shown. The LLM integration accurately calculates the of 13 and 6, resulting in "19" in the left example. In the middle example, it correctly performs subtraction to yield "12". In the right example, the LLM leverages prior knowledge about shapes answer "square."
thermore, in line with the findings from Vicuna baselines (Chiang et al., 2023), implementing con- ditions such "(please output answer text only)" effectively eliminates unnecessary sentences and words from the model's output, thereby enhancing the performance and precision. This was particu- larly crucial for achieving satisfactory performance on VQA benchmarks due to their reliance on edit- distance-based evaluation metrics.
nishing a condition that excludes question-related text proved advantageous. Consistently, Prompt 5 (Table 9) exhibits the best overall performance and therefore used evaluate OCR-integrated LLM baselines. The results presented Table 9 were obtained using 500 validation set samples from DocVQA.
It is worth noting that the results from OpenAI GPT APIs are specific to a certain version at a given time and should be considered during future replication efforts. Our experiments with GPT are conducted in May 2023. As OpenAl APIs actively evolve, updates might affect some trends and sults.
In ChartQA, we found that imposing constraints on answer length and word count proved to be beneficial. We observed that constraining answer length and word count yielded favorable results. The addition of "Answer:" at the end of a prompt significantly assisted the model in executing the QA task. Furthermore, given that LLMs often generate questions as part of their responses, fur-
We also examine concurrent work Latin- Prompt (Wang et al., 2023), which achieves no- table performance Document VQA benchmarks

---

report?
syanamid
'iepariment'
fomaaln usad?
What "chal SIS writte?
central medical department
staritized
150134 BA4822
X
oepuy
Figure 15: Working examples on DucVQA benchmark. The LLM integration model seems to overlook or generate wrong characters when the text within the image becomes smaller and denser, compared to the standalone model.
by processing a system message, which is followed by two conversation turns. Given that LLaVA is designed to generate detailed long output, includ- ing a brief answer example in the initial turn is beneficial.
wi Lairs
Latis
Frequency
2000
6X0
SE0
30.0
1500)
Number OCR tokens
For BLIP-2 (Li et al., 2023), we adhered to the original prompting rules, since they were already optimized for producing concise responses. When OCR is not utilized, we initially input the image, followed by the question to the model. Conversely, when using OCR, the image is input first, followed by the OCR texts, then lastly, the question. While BLIP-2 can answer questions relying solely on the image, the use of OCR was found to be essential for maintaining satisfactory performance.
Figure 16: Visualization of LLM token consump- tion induced by OCR according the presence of' Latin-Prompt. The required OCR tokens (IOCRl) are displayed. used DocVQA tokenizers of Vi- cuna (Chiang et al., 2023).
through a prompt engineering on LLMs. However, several specific conditions are required for Latin- Prompt to function effectively. Firstly, it requires text information of each line. In general, OCR recognizes text and bounding boxes in words, but some OCR APIs provide text and bounding boxes in lines. Latin Prompt requires such line informa- tion. Secondly, using numerous spaces and indents recover layout in OCR results increases input token length for LLMs, as depicted in Figure 16. Applying the method in Wang et al. (2023) entails higher computational costs due to the increased LLM tokens. When using Latin-Prompt with GPT- 3.5, we record an ANLS score of 0.5724.
A.3.3 Cream Prompts
Table 11 displays the queries utilized for address- ing individual tasks during Cream model training. To improve the model's generalization ability, we randomly sampled variety of query types rather than using single query for all tasks. The prompts in Cream training were designed as concise and straightforward sentences as fundamental princi- ple. Prompts for Captioning, QA, and QG tasks were adapted from BLIP-2 (Li et al., 2023).
A.4 Additional Model Training Details
A.3.2 Image-OCR-LLM Prompts
A.4.1 Details of Contrastive Feature Alignment
Table 10 showcases prompts for LVLMS tailored to perform QA task given an image, OCR tokens, and question. LLaVA (Liu et al., 2023a) begins
As explained in Equation Section 3.3.3, we formulate a negative pair relationship even among

---

No.
Prompt
Image OCR Result: foer tokens] / Quaestion: [question]] /
(please oulput answer text only)
(with no more than five words)
Answer:
Image
(please output (oer tokens) "Question: [question]
text only)
(Limit your answer to 50 characters or less)
(Answers should not include question text)
Exact Answer text in OCR Result:
3 Image OCR Result:
toer tokens. / "Question: /question]
ipleuse output unewer text only)
(with more than ten words)
(Answer should not include question text)
[The answer text must be included in the OCR text)
Short Answer:
OCR tokens:
[ocr tokens]
[question]
[ocr tokens]
Question: [question]
OCR tokens:
(ocr tokens]
(question) short to the question
OCR tokens:
(ocr tokens)
[question] A:
OCR tokens:
(ocr tokens| Question: (question) Short answer:
OCR tokens:
[our tokens] Given the imuge, unswer the following question with no more than three words. [question)
OCR tokens:
focr tokens]
Based on the image, respond to this question
short answer: [question). Answer: short possible:
OCR tokens:
(ocr tokens] Use the provided image answer the question: [question] Provide your
OCR tokens:
(ocr tokens) What the
the following question? "[question]" (ocr tokens| The question (question]" can be answered using the image. A short answer is
OCR tokens:
5 OCR tokens:
[ocr tokens) / Question: (question) /
(Please output answer text only)
words)
(The answer word that exists within OCR tokens.)
Answer:
Table 9: Detailed examples of various inference prompts used in OCR-LLMs. This prompts are used in the DocVQA (Tito et al., 2021) test, and Prompt are adapted from InstructBLIP (Dai et al., 2023).
in in-modality features. This approach is rooted in two considerations: (i) even within the same modal- ity, embeddings of different texts objects in im- ages should not carry identical meanings, and (ii) the quantity and the quality of negative pairs sub- stantially impact the effectiveness of CL. When in- modality features act as negative samples, similari- ties should exist in the modality while maintaining differing semantics, thereby providing high-quality negative samples.
negative samples, often consequence of overlap- ping regions between negative and positive samples. Future research may focus on devising methods filter out these inaccuracies from the objective. For instance, while sampling positive pairs, we may set a margin to prevent regional overlaps among chosen samples.
However, for simplicity, we opted for a straight- forward tactic that computes the CL objective with pair sampling strategy (Pani in Equation 1). Our experimental results suggest that the potential false negatives in Equation I are not significant con- cern. Hence, demonstrated in our paper, the suggested CL significantly enhances the model's learning process.
When selecting positive pairs, we select an im- age patch that encompasses the center point of each feature evidence box, and the initial token from the subword tokens of the corresponding auxiliary fea- ture evidence (i..., OCR texts or semantic labels from general objects). From these sampled positive pairs, we use all pairs that do not have the positive relationship to each other as negative pairs. We adopted the standard CL strategy where other sam- ples in the mini-batch serve as negative examples, depicted in Equation 1. This in-batch negative sampling tactic is widely-used recent CL stud- ies (Karpukhin et al., 2020; Radford et al., 2021b).
A.4.2 Cream and LLM Integration Training Process
First, the standalone Cream model is independently trained, prior its integration with LLMs. In the integration training, the weights of the standalone model are utilized as initial weights. The subse- quent sections delve into the specific details of this training process.
Undoubtedly, there a potential to yield false

---

Model
Prompt
LLaVA (Liu et al., 2023a)
You LLaVA, large language model trained by UW Madison WATV Lab. You are able to understand the visual content that user provides, and answer users question using image and natural language.
Follow the instructions carefully and provide answer text only without question included, less than words
HimHumm::
What is the type of image?
(please output answer text only without question and explanation) (with no more than five words)
Assisstttt:: answer document image.
###Humn: [question] [image]
# ##MAssistant:
BLIP-2 (Li ct al., 2023)
(mmaee) Question: [question] Answer:
(image) OCR tokens: (ocr tokens)
Question: [question] Answer:
Table 10: Inference prompts for Image-OCR-LLM baselines.
Step 1: Training Standalone Cream The model training commenced with a large batch size of 384, a fixed learning rate of le-4, and proceeded for 220K steps using 128 A 1100 GPUs. Although not compulsory, the large batch size expedited the loss convergence process. In order gradually progress from simpler to more complex reasoning tasks at this phase, we emphasized text reading and masked text prediction tasks by assigning batch proportions (TR, MTP, Capt., QA, QG) as (22%, 46%, 22%, 5%, 5%). Following this, we began the next phase.
tion utilized a batch size of 192, proceeded for 50K
steps, required 0.5 GPU days using 32 A100 GPUs, and employed a cosine-scheduled learning rate of le-4.
A.4.3
Training Baseline Single-task and Multi-task UDOP
To train UDOP (Tang et al., 2022) under our set- tings, we used the official implementation and the guided training script obtained from the official GitHub repository Since the original paper did not test the ChartQA benchmark, trained the model with it obtain the corresponding result, During the UDOP training on ChartQA, we noted that the validation metric converged rapidly. After 20 epochs (equivalent to 25K steps), ChartQA's score began converging at 60.2. However, we trained it further, ultimately achieving score of 60.7 at around 90K steps.
During the subsequent phase, we modified the batch proportion and hyperparameters for an addi- tional 275K steps: a batch size of 96, a learning rate of 5e-5 with decaying schedule using 32 A100 GPUs, and increased the ratio of QA/QG tasks in the batch. Specifically, this phase was divided into two sub-phases to incrementally increase the QA proportions in the batch. Initially, the proportion (TR, MTP, Capt., QA, QG) was set to (7%, 14%, 26%, 48%, 5%), and the final 60K steps were ex- ecuted exclusively with Document VQA datasets (QA 100%). The standalone Cream model training was completed in approximately three days.
For the multi-task setting, we combined multi- ple datasets and trained the model for 115K steps until achieving a converged validation loss. or- der to conduct our analysis under controlled con- ditions, we employed the same datasets utilized in the Cream's 's training. It is noteworthy that, al- though the original paper also reported results with an increased image resolution of 1024, the model weight, corresponding the high-resolution train- ing, was unfortunately not made publicly acces- sible. We instead used the available pre-trained UDOP with a resolution of 224. Although the high
Step 2: Further Learning to Prompt LLMs
Once the standalone Cream model got trained, our focus shifted toward integrating it with LLMs by leveraging text-rich Document VQA datasets. We observed that the convergence of loss transpired more rapidly in comparison to the aforementioned standalone training, possibly due to both Cream and LLM being well trained. The LLM integra-
ht //gi thub. conm/microooof/i-Cooo Code/tree/ Sain/i-Cod--le

---

resolution could potentially contribute towards im- proved outcomes, noted by Tang al. (2022), the relatively marginal performance gap between the resolution settings implies the resolution of 224, which has demonstrated state-of-the-ar per- formance, remains compelling baseline.
B Contribution of Authors
Geewook Kim led the project as a task force man- ager, initiated the project, and made decisions on overall progress while organizing the research pa- per. Hodong Lee managed the overall dataset construction, co-initiated the project, organized model evaluations, and significantly contributed to code development. Daehee Kim contributed the model architecture with focus on contrastive feature alignment and played key role in craft- ing the manuscript. Haeji Jung managed dataset construction at the project's 's beginning, co-initiated the project, and contributed to its proof of concept. Sanghee Park handled data processing, evaluated off-the-shell LLMs, and made substantial contri- butions prompt engineering. Yoonsik Kim pro- vided critical advice on research direction and de- velopment, heavily contributed the manuscript, and participated in model architecture development. Sangdoo Yun shaped the overall research direc- tion and contributed conceptualization of Cream as a senior researcher. Tacho Kil gave advice on overall research direction and development, and contributed the manuscript senior researcher. Bado Lee advised the project from its beginning, co-initiated the project, and made significant contri- butions to creating the necessary environment and resources. Seunghyun Park advised the project from its inception, co-initiated the project, and sig- nificantly contributed to shaping the project's ' di- rection as a senior researcher.
A.5 Details on Synthetic VQA Dataset
Drawing inspiration from recent VDU literature that utilizes unimodal QA benchmark datasets augment model performance (Powalski al., 2021; Tang et al., 2022), we extended unimodal datasets (Rajpurkar al., 2018; Clark et al., 2020) by creating synthetic VQA datasets called Squad- VQA and TyAiVQA. These were constructed by rendering context pages using WEBVICOB' visual corpus generation tool based on HTML pro- cessing.
To boost the model's information extrac- tion capabilities, we created another synthetic VQA dataset called Wikipedia Key-Value VQA (WKVVQA). WKVVQA consists of synthetic doc- ument images containing key-value pairs extracted from Wikipedia, illustrated in Figures and 17. WKVVQA documents contain key-value informa- tion and synthetic tables.
We generated WKVVQA through the follow- ing process. First, extracted numerous key- value pairs from Wikipedia dump files by selecting HTML tables with either two rows two columns. This basic strategy effectively gathers key-value data. For instance, Venue-EMNLP and y0ar might be identified a table with two columns. Af- ter gathering numerous key-value pairs, we filtered out rare keys and values based on their frequency. These procedures produced a large set of key-value pairs. The key-value pairs then randomly plot- ted white background images. Synthetic tables and card-like objects were automatically generated by simple rule-based manual algorithms.
All participants contributed to this manuscript.
A.6 Complete Dataset Examples
Figure 17 displays example samples of the training datasets. We utilized an array of synthetic and real document images, as well as scene text and general images. While our primary focus lies on processing text-rich documents, incorporating diverse training data proved helpful since context-rich documents often contain figures diagrams.
htattp:://gituub.com///o/////

---

Task
Queries
Text Reading
Read all texts.
Read all texts in the image.
Read all characters in the image.
Given the image. read texts. Given the image, read characters.
MTP
Read masked texts.
Read masked texts in the image.
Given the image, read masked texts.
Read all hidden texts that are covered by the mask area.
Captioning
Explain image.
Use a few words to illustrate what is happening
picture.
Using language, provide a short account the image. Please provide short depiction of the picture. Could you use few words to describe what you percetie in the photo?
Can you briefly explain what you see in the image?
Briefly describe the content of the image.
Provide description of what presented in the photo.
Write description for the photo.
Write short description for the image.
QA
(query)
Q: [query]
Question: [query]
Given the image, answer the following question. [query]
Based on the image, respond to this question with short answer: [query).
Use the provided image to answer the question: [query]. Provide your answer short as possible.
What the answer the following question? "(query)" The question "[query""
be answered using the image.
QG
Given the image, generate a question whose answer is: (answer). Based on the image. provide question with the answer: (answer).
Given the visual represcntation, create a question for which the answer is "[answer)". From the image provided, craft a question that leads to the reply: [answer).
Considering the picture, come up with a question where the answer is: [answer]. Taking the image into uccount, generute question that has the unswer: (unswer).
Table 11: Task-specific queries (prompts) used in Cream training. The prompts for Captioning, QA, and QG tasks are adapted from BLIP-2 (Li et al., 2023).
ITI-CDIP
Webvicob
CC3M
InfographicV QA
OCRVQA
WKVVQA
SAMBURGERS
Tolarcotalk
Mom's Farzly Calendar
VisuaLMRC Pape
Sandin Dugitcon
DEVELOP KOOL
TextVQA
DocVQA
ChariQA
SquadVQA
TydiVQA
Wil/TableQuestionn
DVQA
Poak
Sendg
by
250
28:
142
26:
18
Derahis
Turt
192
26
150
06
174
101 wea 2815.00
Oerata
Figure 17: Training Datasets. This figure showcases samples from a variety of sources, including IIT-CDIP WEBVICOB, Conceptual Captions (CC3M), and Document VQA benchmarks like Doc CharQA, VisualMRC. Wik/TableQuestionn (WTQ), OCRVQA, DVQA, and InfographicVQA. It also features samples from general VQA and scene text VQA datasets. Additionally, it presents samples from our new synthetic VQA datasets, namely Wikipedia Kal-V VQA (WKVVQA), SquadVQA, and VQA.


Reviews:
[{"review_id": "CM9tcGIRQf", "reviewer": "Reviewer_oQuR", "summary": "", "questions": "1. Looking at examples in Figure 14 and 15 made me think about model's capabilities when questions are not asking about the most / the least / the specific type of information. Two examples in Figure 14 ask about the top country or a top item (US and PP). Does the model just really on spurious correlations in answering those? Can it answer 'which country is the third country in terms of the % of immigrants ...'? It should answer France. I think it is important to emphasise it in the paper, or a bit speculate if the model answers questions for the right reasons as the only metric you rely on is accuracy.\n\n2. Lines 259 - 266: Yes, patches and encodings from auxiliary models do have overlapping information, but this information is presented in a different way. Because two models are different and work with different modalities. Is it correct? Then it means that there is no ''direct'' correspondence between features.\n\n3. Also, how do you get the correspondence between image patches and boxes? I don’t get how do you know that visual patch and detected object map? Where does this correspondence come from? Is it annotated? To clarify, you are mapping visual features from visual transformer with features of OCR boxes and object text labels.\n\n4. How did you choose negative examples? Do negative examples still have some sort of overlap with positive examples, e.g. negative patches might cover part of the object, albeit not object completely.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's detailed feedback and comments. In the following, we will address and clarify the weaknesses and questions raised by the reviewer.\n\n---\n\n### On Evaluation Metric for VQA\n\nWe concur with the reviewer's emphasis on diversified metrics - one of the core challenges in the Visual Document Understanding (VDU) research community. We first briefly recap the VDU benchmarks and their metrics: \n\n1. DocVQA [1] and InfographicVQA [2] have been proposed with diverse question types related to an image, which holistically appraise model capabilities. However, we cannot access the test split since they are evaluated via the official competition leaderboard [1, 2] with *confidential test sets*, complying with recent VDU literatures like Donut [3] and Pix2Struct [4].\n2. ChartQA [5] has a public test set. We evaluate VDU models with an exact-match-based accuracy, as conducted by previous literatures [4, 5].\n\nWe propose to include the following discussions on additional evaluation metrics for the ChartQA benchmark with an extended table:\n\n| Table R1    | CREAM-Vicuna | BLIP2xOCR-FlanT5xxL |\n|-------------|----------------|------------------------|\n| Accuracy    | **63.0**      | 18.6                   |\n| ANLS        | **60.3**      | 24.7                   |\n| nED         | **31.4**      | 67.1                   |\n| BERTScore   | **91.0**      | 76.3                   |\n| PPL         | **2.0**       | 24.5                   |\n\nWe emphasize various metrics, including Average Normalized Levenshtein Similarity (ANLS) [1], Normalized Edit Distance (nED), BERTScore [6], and Perplexity (PPL), specified in the review, complimenting exact-match accuracy for model capability understanding. However, Mean Rank is unlikely to be applicable since there is no candidate set provided in the benchmark.\n\nANLS and nED show a reduced performance gap, yet maintaining CREAM's superiority. Conversely, despite multiple metrics indicating BLIP2's poor performance, the BERTScore doesn't starkly classify its responses as nonsense.\n\n\nWe aim to incorporate these discussions in the revised manuscript to thoroughly address the reviewer's points.\n\n\n---\n\n### On the Difference between Cream and Other LLM-Based Models\n\nFrom the perspective of input modality, models in Table 3 can be categorized into three groups:\n\n1. Vision backbone with LLM: visual features (soft visual prompt) are used as inputs to the LLM\n2. OCR with LLM: the OCR results from the image are utilized in text form by the LLM, as evidenced by OCR-GPT and OCR-Vicuna models\n3. Both with LLM: both visual features and OCR results are employed\n\nThe first two types of models were limited in that they use uni-modal information to address visual question answering on text-rich images. The third approach tackles the problem with using multi-modal information.\nOur Cream, featured in the third category, is unique due to its discrete auxiliary encoder for processing feature evidence, e.g., OCR texts (Figure 3).\nIts integration contrasts with models like BLIP-2, which directly inputs OCR texts into the LLM, resulting in substantial computational demands (Figure 7 and Section 4.2).\nCream's advantage is further solidified by our proposed Contrastive Learning (CL) technique, which manages auxiliary information more effectively, displaying resilience against diverse OCR noises (Figure 8 and Section A.1.1). \n\nIn response to the reviewer's comments, we will revise the manuscript to accentuate our model's distinctive attributes and superior performance.\n\n---\n\n### On the Concern over Model Description and Objectives\n\nThe revised manuscript will extend clarity in Figures 2 and 3, detailing the model's architecture. Additionally, revisions will be made to simplify equations and notation for ease of understanding.\n\n---\n\n### On the Functionality of Soft Visual Prompts\n\nIn order to rectify any potential misconceptions, we would like to elaborate on the function of the soft visual prompt. The soft prompts bypass textual conversion; they directly feed outputs from the vision module's output hidden vectors to LLM as a continuous input. Note that these prompts are not converted into text form.\n\nRegarding the doubt about not resorting to soft visual prompts (e.g., reporting bias), we affirm that while simple commonsensical queries might be answered independently by the LLM, complex queries necessitate the soft visual prompt (as exemplified in Figure 1). Note that, evidence of the efficacy of Cream's prompts is apparent in the performance discrepancy between OCR-Vicuna and Cream-Vicuna. It underscores that the soft visual prompt, as created through the proposed Cream model, imparts more valuable and effective information to the LLM compared to using OCR solely as a contextualizing prompt for the LLM.\n\n---\n\n### On the Model Performance\n\nAlthough Cream could not beat all the specialized models, it still holds benefits. MatCha excels in ChartQA but has poor performances in DocVQA and Infographic VQA, as seen in Table 4. On the other hand, Cream efficiently handles varied tasks and shows better OCR noise resistance than methods like UDOP in a controlled multi-task setting (See Figure 8). We underscore that score boosting is achieved without the need for post-processing or ensemble in our unified model. We believe our comprehensive model could lead to future LLM enhancements.\n\nUpon evaluating LLM integration's pros and cons, we noticed better logical reasoning performance but a slight decrease in handling simple OCR tasks or small text interpretation in large images. This is also discussed in Section A.1.3. Given the reviewer's feedback and insight, we would like to highlight these points in the manuscript's main body.\n\n---\n\nThrough this response, we hope to have addressed the listed weaknesses in the review. Subsequently, we will delve into the specific questions raised.\n\n---\n\n### Q1: Can the model answer non-specific questions without relying on spurious correlations?\n\nAs you rightfully mentioned, it is paramount to evaluate models holistically, ensuring they generate accurate responses not merely on the basis of spurious correlations, but also draw on a deeper, inclusive comprehension. We can state that the proposed models indeed possess this capacity. The following examples are correctly predicted answers emphasizing our model's capability against spurious correlations:\n- Q: Which country is the {first / fourth} in terms of the percentage of Sub-Saharan African immigrants? A: {U.S. / France}\n- Q: Is the sum of the smallest two bars greater than the difference of the largest and 2nd largest bars? A: Yes\n\n\nIn our research, we utilized representative benchmarks including ChartQA [5], InfographicVQA [2], and DocVQA [1]. These benchmarks encompass an array of queries, from relatively simple to complex in nature, like queries necessitating the identification of a second or third highest position or those necessitating arithmetic operations. Addressing these text-rich benchmarks cannot be achieved solely via reliance on high-frequency correlations.\n\n\nUpon considering the reviewer's comments, we are eager to enrich discussions on evaluation metrics with the augmented results as discussed earlier in the response to the point ``On Evaluation Metric for VQA’’ (Table R1 refers). Additionally, we would like to refine Figures 14 and 15 to better exemplify the models' capabilities.\n\n---\n\n### Q2: On correspondence between different modalities\nIn response to the point, the CL objective doesn't indicate that two distinct features should have a direct relationship or identical vector values. \nThe CL objective recognizes that certain feature pairs (i.e., positive pairs of patches and subwords) maintain more similarity to other features and incorporates this understanding into calculating the objective.\n\nIn the paper, we found that the proposed CL objective significantly contributes to increasing the overall performance and robustness to OCR noise. That observation again emphasizes the merit of our approach. By bringing the Vision and OCR features into a subspace, we create an organic interaction between these modalities, promoting a holistic understanding of the given tasks. \n\n---\n\n### Q3: How is the correspondence determined in CL and how to prepare the correspondence?\n\nIn CL, correspondence is established based on the position of a visual patch and its associated feature within an image. We make the connection of each visual patch and its feature from specific image regions, such as the bounding box of an OCR text.\n\nTo prepare the detection results, we utilize off-the-shelf OCR and object detectors to produce pseudo labels if the original training dataset does not include them. This is a common practice in the VDU field [3]. Further information on our dataset samples is available in our supplementary materials. Details about the detectors used can be found in Section 4.1 and A.2. \n\n---\n\n### Q4: How were negative examples selected in CL and is there a chance for false negative samples?\n\nTo generate negative examples, we utilized a conventional method in which the other samples in the mini-batch function as negative examples, as demonstrated in Equation 1 of our manuscript. This approach, also known as ``in-batch negative sampling’’ is commonly used in the field of CL as indicated in references [7, 8].\n\nUndoubtedly, there is the potential to yield false negative samples, often a consequence of overlapping regions between negative and positive samples. Future research may focus on devising methods to filter out these inaccuracies from the objective. For instance, while sampling positive pairs, we may set a margin to prevent regional overlaps among chosen samples.\n\nIn this study, for simplicity, we opted for a straightforward tactic that computes the objective with a sampled subset of pairs instead of the entire set ($P_{\\text{uni}}$ in Equation 1). Our empirical evidence suggests that most of the time, false negatives are not a significant concern in the objective computation when the full set isn't in use. As a result, as demonstrated in Figure 8, the proposed CL significantly boosts the model's learning process. Based on the feedback, we would like to elucidate CL's role further and incorporate these discussions into the revised manuscript.\n\n---\n\nOur response and the suggested revisions will address the raised points and further enhance the quality of the paper. Additionally, we are open to further discussions and will gladly respond to any more queries or doubts raised. Considering the reviewer's inputs, we will eagerly revise our paper to contribute better to the research community.\n\n---\n\n### References\n\n[1] Tito et al. ``ICDAR 2021 Competition on Document Visual Question Answering.’’ ICDAR, 2021.\t\n\n[2] Mathew et al. ``Infographic VQA.’’ WACV, 2022.\n\n[3] Kim et al. ``OCR-Free Document Understanding Transformer.’’ ECCV, 2022.\n\n[4] Lee et al. ``Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding.’’ ICML, 2023.\n\n[5] Masry et al. ``ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.’’ ACL Findings, 2022.\n\n[6] Zhang et al. ``BERTScore: Evaluating Text Generation with BERT.’’ ICLR, 2020.\n\n[7] Karpukhin et al. ``Dense Passage Retrieval for Open-Domain Question Answering.’’ EMNLP, 2020.\n\n[8] Radford et al. ``Learning Transferable Visual Models From Natural Language Supervision.’’ ICML, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting idea that is definitely relevant in multi-modal NLP: how to combine different information sources (multi-modality) and, perhaps, use it along the large LLM for some task. The datasets are challenging as they require some sort of reasoning over charts, etc., so this is something that needs more attention in the field.\n\n2. Possibly plausible way to align features from different modalities.\n\n3. The results hold across multiple datasets and (more or less novel) models and propose novel insights for those models.", "reasons_to_reject": "1. Evaluation of the models is done with accuracy (I believe). Accuracy is known to be not that good for VQA tasks as it does not tell us whether models answer questions because of spurious correlation or for the right reasons. I think a different evaluation metric here is needed to inspect that (related to my first question to the authors in Questions For The Authors below). Perhaps, mean rank on different question types or perplexity would tell how bad the model is actually.\n\n3. Also, as both proposed model and models that were proposed before are generally bad on this task (acc to Table 3), what are the exact differences between Cream and previous approaches? What makes Cream better? Is it its more modular nature? This was missing and I think this is crucial to frame the proposed idea.\n\n2. Model description, training objectives and evaluation are hard to understand. Reading these descriptions is a bit complicated, I got the idea, but it took me several reads to fully grasp what was happening there (which implies that the whole approach is technically quite complex and involves many elements).\n\n4. As I understand, when generating soft visual prompts, the LLM predicts an answer only on text produced by Cream. Although this text is generated based on multiple modalities, how big is the effect of LLM in answering the question correctly or incorrectly? Does it really really on the output of Cream or it just has a lot of knowledge about things and often outputs the correct answer (e.g., reporting bias). Plus, comparing Table 4 and 3, it seems like standalone performance or previous approaches is often better than Cream + Vicuna 7B. For example, 63.0 for ChartQA for Cream-Vicuna7B, but MatCha gets 64.2.", "questions_to_authors": "1. Looking at examples in Figure 14 and 15 made me think about model's capabilities when questions are not asking about the most / the least / the specific type of information. Two examples in Figure 14 ask about the top country or a top item (US and PP). Does the model just really on spurious correlations in answering those? Can it answer 'which country is the third country in terms of the % of immigrants ...'? It should answer France. I think it is important to emphasise it in the paper, or a bit speculate if the model answers questions for the right reasons as the only metric you rely on is accuracy.\n\n2. Lines 259 - 266: Yes, patches and encodings from auxiliary models do have overlapping information, but this information is presented in a different way. Because two models are different and work with different modalities. Is it correct? Then it means that there is no ''direct'' correspondence between features.\n\n3. Also, how do you get the correspondence between image patches and boxes? I don’t get how do you know that visual patch and detected object map? Where does this correspondence come from? Is it annotated? To clarify, you are mapping visual features from visual transformer with features of OCR boxes and object text labels.\n\n4. How did you choose negative examples? Do negative examples still have some sort of overlap with positive examples, e.g. negative patches might cover part of the object, albeit not object completely.", "paper_topic_and_main_contributions": "The paper proposes an information fusion approach to tackle the problem of visual question answering. Authors use multiple modalities (objects, labels, patches, OCR detections) to learn to answer questions about image content accurately. They propose a way to fuse different modalities, bridge them together and use this information to either directly predict an answer or generate a prompt used by an LLM to predict an answer. The paper describes an information fusion approach together with how it's used in model's training. Authors conclude the paper with evaluation against previous related approaches and some analyses showing that this information fusion approach works well.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pimBA8VmYy", "reviewer": "Reviewer_h5KP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the efforts to provide valuable and insightful feedback. The reviewer's recognition of our methodological approach, rigorous evaluations across diverse visually-situated language understanding tasks, and our commitment to reproducibility greatly encourages us. We would like to address the raised concerns below. \n\n### Ablation Study\n\nTo address the weakness raised, we would like to draw attention to the ablation study presented in Figure 8, where the efficacy of Contrastive Learning (CL) is quantitatively examined under different levels of OCR noise. As demonstrated, a model utilizing CL shows significant robustness, especially in high OCR noise environments, highlighting the integral role that CL plays in improving resilience.\n\nAdditional detailed analyses focusing on the effect of CL and OCR are provided in Appendix A.1.1 and A.1.2, Figure 11, and Table 5. These analyses further delineate the contributions of CL when Cream is subjected to OCR noise and auxiliary information noise.\n\nRegarding the PCA analysis in Figure 9, it offers a crucial supplementary view of our findings. While the practical difference brought by CL might not be explicitly visualized in the PCA, it provides an understanding of the resultant feature alignment in the representation space when CL is applied.\n\nIn light of the reviewer's inputs, detailed discussions on CL and OCR in the Appendix will be relocated to the main text in the final revision for reader convenience. It is hoped that this response has clarified and enriched the reviewer's comprehension of the paper. As we undertake the final revision, we commit to addressing improvement areas and enhancing the value of this work for the broader research community. We extend our gratitude for the reviewer's valuable feedback in refining the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces the novel Contrastive Reading Model (Cream) with a unique methodological approach to enhance language-image understanding. \n- The authors have made methodological strides in handling in-modality features and crafting positive and negative pairs, backed by rigorous evaluations across diverse visually-situated language understanding tasks. \n- The analysis, coupled with a commitment to reproducibility by releasing their codebase, stands as a testament to the paper's thoroughness.", "reasons_to_reject": "The paper falls short in providing an ablation study, which is crucial to elucidate the quantitative contributions of each introduced component. For example, even with the PCA analysis in Figure 9, the absence of a detailed examination of the contrastive learning technique leaves unanswered questions regarding its impact and significance. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper enhances Large Language Models (LLMs) for applications in the visual domain, particularly for understanding text-rich images. Recognizing the limitations of current models in this arena, the authors introduce the \"Contrastive Reading Model (Cream),\" a novel neural architecture. Cream integrates both vision and auxiliary encoders, reinforced by a contrastive feature alignment technique, to augment the comprehension of language information within visually situated contexts. This approach not only bridges the gap between vision and language understanding but also paves the way for sophisticated Document Intelligence Assistants. Through comprehensive evaluations, the paper underscores Cream's impressive performance, positioning it as a significant advancement in visual document understanding. The authors further contribute by sharing their codebase and datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dcznB9QpvR", "reviewer": "Reviewer_ZnEz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's time and effort in evaluating our paper and providing expert insights. The positive review of our paper is truly encouraging to the authors. We are grateful for the reviewer's valuable feedback and endorsement of its value. As we proceed with the final revision, we will diligently strive to address any areas of improvement and further enhance this work's value to the research community. Once again, we sincerely thank the reviewer for the reviewer's thoughtful evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel Model: The paper introduces a novel neural architecture called Contrastive Reading Model (Cream). This new model has the potential to significantly improve the language-image understanding capability of LLMs in the visual domain, addressing a critical limitation of existing LVLMs.\n\n2. Addressing a Practical Problem: The paper addresses the limited application of LLMs in the visual domain, particularly in text-rich visual tasks. By proposing Cream as a solution, the authors bridge the gap between vision and language understanding, leading to the development of more sophisticated Document Intelligence Assistants.\n\n3. Experimental Results: The paper presents compelling experimental results demonstrating the strong performance of Cream in visually-situated language understanding tasks. This empirical evidence reinforces the effectiveness and practical applicability of the proposed model.", "reasons_to_reject": "I think this is a good paper and should not be rejected.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a novel neural architecture called Contrastive Reading Model (Cream) that enhances the language-image understanding capability of Large Language Models (LLMs) in the visual domain. The main problem addressed by this paper is the limited application of LLMs in the visual domain, particularly in text-rich visual tasks. The authors propose Cream as a solution to bridge the gap between vision and language understanding and develop more sophisticated Document Intelligence Assistants.\n\nThe main contribution of this paper is the introduction of Cream, which combines vision and auxiliary encoders, fortified by a contrastive feature alignment technique, to achieve a more effective comprehension of language information in visually situated contexts within images. Cream overcomes the limitations of existing LVLMs by handling fine-grained features without missing image details. The authors demonstrate the compelling performance of Cream in visually-situated language understanding tasks and position it as a prominent model in the field of visual document understanding.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce the Contrastive Reading Model (Cream) designed to enhance text-only LLMs for visually-situated language comprehension. In particular, Cream aligns visual features from both image patches and OCR texts through a contrastive objective. The aligned features are subsequently encoded as soft prompts, enabling LLMs to interpret visually-situated language found in documents, charts, and infographics.

The unanimous consensus among reviewers is that Cream offers a novel approach to integrating visually situated language into text-only LLMs. Although there were concerns regarding the rigor of the experimental setup and ablations, I believe the authors adequately addressed these in their rebuttal.
Decision: Accept-Main